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FOREWORD 



This Handbook is one ot several major GAEL assessment reports based upon 
GAEL Working Papers produced in 19^5. GAEL followed an unusual developmen- 
;tal plan in validating its initial work and preparing these reports. In order to insure 
that its deyelopmental work was fully reviewed and tried out prior to final publica- 
tk)h, the GAEL vyorking Papers were used experimentally in approximately 80 
CAEL institutions during 1976-76. In a formal review procedure users provided 
criticjue^arid suggestions that the authors used in preparing this revision. Validation 
of these current materials also included^a series of field studies that addressed 
irriportant issues concerning reliability, validity, and the value of assessment to the 
student. Those validation studies are reported in The CAEL Valida^on Report, 
1 976; aspects of the findings especially pertinent to content are sunfimarized briefly 
' in Chapter III of this Handbook. The Prefage^summarizes the more important 
issues with which the authors tried to deal in their revisions and the major changes 
that distinguish this Handbook from the earlier version, originally published as 
Working Paper No. 10,- 7/76 Use of Expert Judgment in the Assessment of Expe-^ 
rier)tial Learning. 

Warren W. Willingham 
GAEL Project Director 
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PREFACE 



t 



Thelndividualized-and often highly unique nature of experiential learning poses a 
number of problems fQr the faculty members and administrators charged with the 
assessment of that leamlng. Standardized tests,- for example, are usually In- 
• adequate or Inappropriate. Assessment must, therefore, be placed in the hands of 
an,"expert,'* someone who has specialized knowledge and experience In a relevant 
discipline or field. This Handbook Is an attempt to familiarize those involved /In 
assessment with some of the basic principles and practices in the application of 
expert judgment. Thpugh^others may find this book useful, it is primarily Intended for 
faculty' rtiembers, administrators, and consultants Involved in assessing the 
exporientially-based learning of students who aro seeking academic credit for that , 
teaming. ^ ' 

Responses to (he working draft of this Handbool< suggested an extremely 
diversi audience, ranging from educational philosophers to engineers. Con-' 
sequently, an effort has been made to avoid overly technical language and wher- 
ever possible to illustrate and clarify points with concrete examples. Those farpiliar. 
with the working draft of this docurnent will note several changes. The introduction 
rK}w contains a more exienslve.discusslon of the problem of choosing an expert 
judge as well as a brief overview of some basic principles. The two separate 
chapters on interviewing have been combined into one chapter with a consequent 
reduction in redundancy and what is believed to be a baiter organization. A brief 
. •'Concluding Note" tjiat reminds the reader of some general assessment principles 
tttis replaced the chapter on "Use and Evaluation." * * , 

It Is hoped that, this Handbook will be useful in three ways: Firsts practical 
methods and procedures are suggested that can help institutions make moje.^' 
effective use of expert judgment. Second, some examples of approaches fo as- 
sessment are presented that would, with some modifications, be suitable for appli- 
^ cation a* an institution where experiential learning must be assessed. Finally, it is 
hoped th^it the principles and methods suggested here will help institutions to 
evaluate the quality of their assessment on a continuing basis. 

The writing of this Handbook was the joint effort of many people. Richard R. 
Reilly, an Educational Testing Service staff member through most of the project 
and currently at American Telephone and Telegraph Co., prepared Chapters I, II. 
and the doncluding Note, and contnbuted to Chapter III. Ruth Churchill of Antioch 
College wrote Chapter IV, and Arnold Fletcher of Thomas A. Edison College 
co-authored Chapter V with John J.. D. Clark of ETS, who was on leave at Thomas 
A. Edison College when the wo;king draft was written. Chapter VI was prepared by 
Myma Miller, who was at Empire State Collb^ft when the writing began and Is now 
. with VemTiOnt State Colleges. Judith Pendergrassof Metropolitan State University 
was the senior author ^of Chapter III. Jane Porter Stutz of ETS contributed to 
Ch apter III and coordinated commumcations and the compilation of the manuscript. 

The following people reviewed drafts of the Working Paper or the revised draft 
and made manV/Helpful suggestions: 

Thomas Donlcn, Educational Testing Service* 

Sheila ^ordon, LaGuardia Community College 

Joan Knapp, Educational Testing Service 
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Robert Ramos, American Telephone and Telegraph Company 
i'^ ' Ri*6rt Sexton, University of Ke^^^ 

"\ . ;D«irte TIppo, Warren Wi^^ ' 

In a<WW(^^ arid administrators from many GAEL institutions completed a 
vf ' questk>finalre on the Working Paper and provided useful recommendations. 
I' Rnally, appreciation Is expressed to Lorraine Simon, who typed the early 

5 drafts^aryJ helped in countless other ways during the project. 



. ' ' - . ' Richard R'. Reply 




■ iRtrcduction 



Richard R. ReiOy 

_ The assessment: of (earning Invariably requires humeji judgment. This is clear 
enough in the case of a college instructor who must assess the learning of students 
taking a course. Perhaps less clear is the extensive reliance on judgment in the 
d6vek>pment of so-called "objective" multlple-choice achievement tests. In the first 
instance, me instructor may use direct observation of oral and written performance 
as a basis for grading istudents. In the second instance, judgment is involved in the 
selection of content to be covered and in the writing of items. In both of these 
exanhplest^ person making judgments Is presumed to have some special know- 

^ ledge or expertise in the relevant subject-matter field. 

/^-The assessrnent of experiential learning certainly Is no different in demanding 

• expert: judgment. .Whether tlie learning has .been derived from a sponsored or 

' ndnsppnsored experience, judgnient will be necessary to evaluate the amount and 
quality of what ' has beenjeamed. In many instances particularly when prior, 
Qohsponsored learning is being assessed, assessors are forced to rely on only a 
limited ariidurit of information. It is especially important, then, that the assessors, in 
addition to being "experts" in a particular substantive domain, also t)e competent 
judges; Experts who are not properly prepared to conduct assessment jeopardize 
the equity of assessment for the institution, and more importantly, for the individual 
student. Competent judges, on the other hand, while not guaranteeing perfect 
assessment, can ensure an equitable and consistent treatment of all students. 

Overlap Between Different ^Methods 

Four major assessment techniques are discussed in this Ayandiboo/c-Hnterviews, 
product assessment, performance assessment, and assessment of written mate- 
rial. These are not the only ways in which to assess experiential learning, but it ts 
believed that these four methods Comprise the most useful and relevant approaches 
for applying expert judgment, though separate ch&>pters are devoted to each 
method, it will biecome apparer^l to the reader that areas of overiap exist across 
methods. To some extent distinctions between methods are arbitrary and involve a 
matter of emphasis rather than a clearcut demarcation. This is perhaps most 
obvious in the oase of product assessment and performance assessment where 
some of the methods and suggestions can be used interchangeably. The emphases 
are decidedly different, however. In.product assessment there is greater concern 
with what Is produced and less y^Whhcw lus produced, whereas the reverse is true of 
performance assessment. The fact that there is overlap is not necessarily undesira- 
ble. It will be found that for many assessment problems a combination of two or more 
procedures may be much more effective than a single method. The principles of 
product and performance assessment can be applied in tandem tn those situations 
where both the process and the end product must be considered to yield a valid 
assessment. Likewise, interviews may provide a useful adjunct to other assessment 
methods in many situations. It should be made clear that the separate treatment of 




the four assessment methods in no way implies that each method should always be 
lised in isolation. Indeed, it Is the authors' intent and hope that users will recognize 
the advantages of applying more than one technique. 

Purposes of This Handbook 

ThisHandbook has four major objectives. First, it is hoped that readers can develpp^^ 
Hotter understanding of some of the general methodojogical problems and issues 
related to the use ot exp>ert Judgment. Issues such as reliability and validity should be 
understood^d considered before applying any assessment technique. 

A s^nd objective is to provide the reader with specific practical suggestions 
related to each of four common applications of expert judgment. The discussion of 
the pfoblenris and pitfalls of interviewing, p:oduct assessment, performance as- 
S3ssment, and the assessment of written materials should be helpful to those 
engaged Jn the application of these methods. 

A third objective is to present the most impo'larit principles of soiind assess- 
ment as a series of steps or checks to follow in conducting assessment. The general 
principles are presented later in this chapter and reoccur in modified form at the end 
of each chapter dealing with a specific type of assessment method. 
, A final objective is to broaden the reader's awareness of various possible 
approaches to assessment through the presentation of practical examples and 
prototype^procedures. Some readers may find that an adaptation of a suggested 
procedure meets a local assessment need. 

Choosing Assessors ^ . 

Most of the content of this Handbook a ssumes that assessors have been identified 
and assigned. For many assessment problems, however, the task of choosing 
appropriate assessors is a cnticai and difficult one. Whitaker has distinguishec) six 
groups of potential assessors. (1) student learners in sponsored programs, (2) prior 
learners seeking credit for unsponscred learning, (3) faculty teachers, (4) nonfacuity 
teachers or supervisors in a sponsored learning program, (5) participant-observers 
such as clients or co-workers. (6) outcome observers. This last group inclbdes ' . . . 
persons who have not participated in the learning process in any way, but are called 
upon to participate in the assessment of learning outcomes Unlike the other 
groups, outcome observers have no association with the learning process that might 
make them convenient and natural choices for assessors. This will occur most 
often with the assessment of pnor leamtng where a person with expertise in a 
specific area must be choseri to conduct an assessment. 

Norn(ial!y, choosing an outcome obser\'er is a two-stage process involving 
Identification and selection. In the identification stage two or more assessors are 
identified as having expertise m the subject matter field for which credit is sought. At 
this point it becomes necessary to raise a very basic question. Who is an expert? In 
response to this question we offer a working definition. An expert an individual 
having special skill or knowledge denved from expenence. education, or training. 

— \- 

'{JftM Gl VfikWw. A5$«sso(S and Vnot OoaH^tons. «n Morr/s 7 Ke«ton ard A&vxsates. tip^9nbat s^earnng 
RitcnMi^. Chtfactcftstc9. 9nd Assessment San Franc^soo Jossey-Bass. t976. p i93 




The definition purposely ckjes not imply any fomial academic credenlials" so that 
Individuals who have expertise because of their expenenlial learning (eg. an 
avocaticnal pursuit) c^be included. 

How, then, Is onelPldenlify an expert? Specific types of evidence that might be 
used to delermtrie whether an individual is an expert in a gr/en subject or field 
Include the following: 

1. Recorpmendations from two or three other experts m the same or a closely 
related field. . 

2. Published works (e.g., journal or magazine articles, books, etc ) in a given 
subject. ' 

3. Technical products or artistic works in a related area. 

4. A directly related formal academic degree. 

5. .Other formal credentials such as a license. 

6. Awards or honors for achievement in the field. 

7. Instruction or teaching experience in the area. 

8. Supervisory experience of individuals working in the same field 

9. Membership m organizations or societies with a focus closely related to the 
subjed^area. 

Although there are really no hard and fast rules fpr identifying experts, no single 
one of the foregoing suggestions should be relied upon solely As many mdepende'^t 
sources of verification as possible should be obtained. 

Assuming that several experts have been identified, usually one expert is 
selected^ to conduct the assessment. Wha! are some of the considerations that 
govern which expert is selected? Whitaker offers some useful criteria, which he 
refers to as essential assessor characteristics, "The first of these characteristics is 
subject-mattet expertise. Although all of the potential assessors identified will have 
some subject-matter expertise, some may be stronger m certai.T areas than others 
The strengths ^nd weaknesses of the potenlal assessors should be judiciously 
considered in terms of the spscific learning bemg claimed ^e g . theoretical know 
ledge vs. practical skills). 

A second criterion involves the psychometric expertise of the assessor 
Whitaker defines psychometric expertise as . . secondary knowledge of the 
assessment process that js sufficient to enable one to select and adapt techniques 
and instruments that others have developed and perfected Increasing the 
psychometric expertise of assessors is precisely what this HarxJbook i*^ about 
However, the experience and knowledge of the expert with regard to conducting 
assessment should be taken into account dunng the selection stage 

A third characteristic of effective assessors is what Whitaker refers to as 
familiarity with case data. Outcome observers as a group are typica", not very 
familiar with ail aspects of the case being assessed In many cases, ''ov/ever. 
^.ssessors will know something about the learner, the teacher (if any), the learning 
process, or uther aspects of the case at hand In genera., such knowledge will help 
make the assessment more valid, ©nd this should be considered in selecting 
assessors. 

'ABhou^ < a ajways (iesirao^ » nave moarp*« assessor « *» * ecogn/ed fai >' /Tx^a trvs bMj^t a* 

costs or other practical considers! orts ^ 
n^Uker. pp 198-199 
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JiiQ fi(>al two criteria ihsd should do cofv^cjered are !r>e ofe/eccrwiy^and/potrva 
pon of tho as«5&sof. TTro o^port /udg© should fee froe of pfOjudice lowmc lear no:. 
.,tho and any olhot aspects of the Jeamiog pfoco£»& and, m addibofvshoutd 

M hJflHfy nwBvaled to conduct assessment x. the most thorough and oquttabie 
'rpahnerpossiWo* - \ 

Not an BSSassors sefocted will posse:>s an of tl^ese chaiactonstics, of course, 
fndaad, jf it ware aiways possible. lo seioct asse^sois meeto^ ah ol tnesa cntena 
thOfO wouki be oo need for a handbook ilKe lh<;b one. Howev&i . warefu^ ^entiricatior. 
and saloction of o^pert judges can cons^derabty simplify the task of prepanng 
assasisom. 



Onganizatlonand Scopo 

The other chapters of the Handbook present a great dea. ^ ^nloimal*or., as w^li as 
materials* that ipght bo adapted fot use in assessment Chap q\ \\ deats wtth some 
of tho basx^ te£hnica^ and melhodoK)g(Uii wonside rations ^mmun to aii judgnientat 
rpalhods of assessment Since tho mtei .chapters assume some famittarity ^th 
these sor^rat cooceptjj^ <s recommended that Chapit?: «« t>e read before an> of the 
chapters dealing yrfth'^dfic techniques. 

Chapters. Ill throt^h Vt discuss speafic assessment te^hntquss. Chapter II! 
discusses tho device most wcmmoniy usee ic assess e^iporientiai earning -the 
interview. Hie (fifforent types ol interviews arc desait>ed. potentia. prot^ms are 
discussed, and a vanety of piactica* suggestions are ."nado foj the improvement of 
mtervkwmg and related techniques. Finally, the suops *n ^vo^ng a siaiclured 
intervtow are presented and ^Di^tfatoa Chapters and » discuss the methods of 
productand perfora*anca assessment" Bune cornm?,)! . piot«ems are descnbed and 
si^estiona f^ improvement are offered Both chapters deta^t mudOk procedures fo^ 
the cor^i^.dl ^assessment because of th^ hea^^ .ate :hat writer, materfa^ ptays ^. 
^ experuritmi loarnrng, a separate chapt&i has beer, ^nuudeo ivx discuss some of the 
relavfjant issues* probiorns. and cnotrKXis. Chapter » * aiso *.unta*r)s a vauety of usetui 
suggestions that might bo a^-^"^^ to assessment otvynttonmatena. A Concluding 
Note^ summari;:es si\ ba^u assessment steos aiiC r>ow the> ^pu\c be appbod to 
ttte technk^ues presented m earlfor chapters 

An Overview of Some General Princlplosr 

Because so much emphasss »s placed on a soi bas«. as:>essmQnt punc^ptes *n the 
Handbook, it may t>o he^pfiii fo^ trie reader to t^ioH) voa^^dal the gcnofai steps 
recommended at the end ot Chapter ii lo .oxistrato the conHnon oiernonts of an 
assessment prcAiems, and ai the same l.mcv provide ar. ovenrrew of some voSic 
jxmoples, the s*«. basic steps of assessment are presented together with s^x ;ec 
ommended stops tor assessment of sponsored off carnpus learning <n Tabkj 

Tho first step *n :he assessmoni of sponsored loarmng *s Ptogtam Definition, 
which has no genera^ countetpart smce for a .argo group of expenentiat leaif>ers. 
namely prior learners, no set progiarr. e»ei e^^sied Tho first gener a^ step. Cstabhah 
Cfkenon Stanaartss, does have a d.ect <x)uniorpaft if- Speaficat^or. o/ Lsarrung 
Oufcomes In both cases p«nnif>9 <^awn exact?> »vhar ^ to be assessed *& cntk^ai ioi 
effecave assossmor^i 
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Table 1 , , 
comoarlson of Genera/ Steps Recommended for Assessment of 
"^x^SSriL/ng w/tf, suggested Steps for /Assessment of 
Sponsored Learning 



GoneraJ Steps for ConcJudir^g 
Assessment 



I, HstaWJsh Critorton Standards 

\l Setoct and Stmctu^e Assess- 
ment 

Ul Plan the Administtation of As- 
sessment 

IV. Bplate Judgments to Ot^erva- 
tions 

V Record and Report 

VI Monitor Assessment 



Steps <of Assessment ot 
Sponsored Learning 

\ Program I>etinition 

II Spedftcation of teaming 

Outcomes 
\\\ Estabhsh the Assessment 

Procedure 



IV Formattve Assessment 

V Summaiive Assessment 
vl Evaluation ol Assessment 



Reco^ an0 R.pon ^« J^fl jlt s^^^^^^^^ Relating judg- 

Assessment which are the next ^.o ste^^^^^^^ 

ments toobEervatK>nss.mply means natas^.sofS^ 

closety as pos-blo to the ev dence at ,he assessor 

be checked for major Haws or ^^akness^s ^^^^^^ „ 

ThefolWChaptorsoHercons^derattea^^^^^^ ^^^^^ 
.«; hooed that the reader wirt recognize thai effective asse-s w 
prrS^mgardless C the technique or method t^emg app-ed 
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H^nTH^^y^^^^^^ ^"^ Related 

To The U3e Of Expert Judgment 
■ in Assessment 

Richard R. Rellty 

set of general prind^es^L^ffrti^^ T'T '° ^ ^^^sonably well-defin«J 
search coni'whaf«^^ °' 'Ws re- 

that such erro^ ha^e "ci «^rs,udTn? 'rTf T.'^''^' 

sors to be aware of of ih^mr^" " 'T^^"' administrators and asses- 

tal assessS '""^^ "'^'""'"^ ^"'^ surrounding {udgmen- 

do aSlrt'Sld'firlt^' ^"'^^^ 'yP'"'^ "^^ke ^nd what to 

Injhe prS; c^ap^^^^^^ °' •'^^ '"'^Se in assess.nen 

Judge do fXesS?^ 7^?»« ""'"^^^ "What does the expert 

surement: (5%alua.ioi a^'^ns^^^ documentation; f4) mea- 

experience hsibeemdmiSdlndr^J^T ? ^ ^"'"'^^d through 

Judgment can 6» applied tomeasure the nTirr?,L ^" ^'^P^'* 

fo era/ware whXe^ that . ^"'^ °' '^'^ "^arniig and then 

concise^ des^ShgK . 'g ^'022"^'"^ 

some cases, the expert play^rrale in Z^^l? "''"'^"P' <''«''^'^^'P''>'ff)- In 

identifying teaming Eamorof ^ , ^ ^""^^ '"'9'^' ''^ '"^ol^ed m 
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highly specialized, but the assessor should be able to" make them explicit and put 
them Into objective terms. 

S«lectlng. In some cases the assessment procedure may be predetermined by 
the institution or chosen by the student, as m the case of a product presented by a 
student for credit. In many cases, though, once the criteria have been specified, 
the expert will decide which type of assessment procedure is most appropriate for 
eliciting the most relevant sample of behavior, performance, or other evidence 
The expert may. in fact, be able to take advantage of existing assessment proce- 
dures that could be adapted and structured for a student with relatively little revi- 
sion. 

Structuring. The degree of control over the structure of the assessment process 
that an individual assessor has will vary depending upon the situation Structure 
may be imposed by others involved m the assessment process, for example But in 
many situations the assessor can dec.de whether the assessment will be highly 
structured (e.g., an oral examination) or relatively unstructured (e.g , a loosely 
patterned Interview). 

Adapting. The expert assessor can have considerable impact on the validity of the 
overall judgments made by adapting the assessment technique to the experiences 
and needs of the indwiOudl. Adaptation does not necessarily imply a lack of struc- 
ture, but rather the selection of the fairest and most relevant information for the 
demonstration of a student's particular learning or competence 

Observation. The expert must observe a students behavior, performance, or 
products before reaching a decision. The observations arc the critical stimuli to 
which (he expert makes a response tn the form of a judgment or assessrrent The 
competence with which an expert can make observations is determined by some 
factors beyond the contro: of the immediate as^essmept process, such as experi- 
ence and intelligence. Emphasis on the other functions of the assessor's role can 
go a long way toward enhancing the quality and relevance of the observations 
made. Focusing tlie assessors attention on the specific behaviors or the most 
relevant aspects of performance can be dont with clear, concise definitions, for 
example. 

Recording. Iri instances where assessment is relatively complex or where judg- 
ment occurs sometime after observation, it is extremely important that some record 
of observations be kept. Observations may be recorded electronically, by the 
taking of notes, or only by the central nervous system of the expert. 

Judging, The most critical funct.on of the assessor s role is. of course, the act of 
judging 01 quantifying the student s learning. It is withm this aspect of assessment 
that the expert must somehow cur.sider what evidence has Deen presented, elimi 
nate what is irrelevant, weigh what is relevant, and finally balance this against 
some standard of competence. The act of judging itself, though certainly crucial, 
does not take place in a vacuum. The other functions of the assessor s role in 
helping select the relevant evidence or sample of performance, in structuring the 



assessment process, etc., are of equal impcrtance in minimizing the errors that 
may be made In judgment. 

These seven functions of the expert assessor's role may not all be carried out 
by the same individual, nor is it necessary that all seven be performed with each 
r«w assessment. For some common assessment situations the functions of crite- 
rion definition, selecting, and structuring may be performed only initially, since the 
results: wiir have generar application for all students seeking the same type of 
credit. For highly individualized an J unique leamlnc) experiences it may be neces- 
sary for alj seven functions to be performed on an ad hoc basis with each new 
as^^sment. 

THE PROBLEM OF ESTABLISHING STANDARDS AND LEVELS OF 
CdMPETENCE 

Because assessment can be viewed as a comparison between the evidence pre- 
senteci by a sti^dent and some fixed standard, it Is essential that the assessor have 
a clear understanding of what standards are being applied for a given assessment. 
In a measurement sense, the simplest type of assessment would be one in which 
the student's evidence is compared against a standard and a decision is made as 
to.whether the evfdence meets the standard or not. In many instances, however, 
the assessment process calls for a finer discrimination. A student may be awarded 
from zero to ten credits, for example. In such cases more than one standard is 
ot^viously needed, and, in^fact, a continuurp with anchoring standards at various 
points along a scale may be implied. Some of the most critical kinds of errors made 
in judgment have been shown to result from a lack of well-defined standards or 
criteria. Failure t6 provide specific standards can create at laast two different kinds 
of problems. First, an assessor may have a very specific set of subjective stan- 
dards that may not be shared by other assessors. This results in inconsistency in 
the assessment process since the final judgment made may depend more upon 
the assessor's unusual standards than upon the relevant evidence presented. A 
second possible response to a lack of clear standards is that the^assessors may 
leave the standards vague. Under these conditions assessors v;ould be more likoly 
to '*play it safe" and never really commit themselves strongly In making judgments. 
This too, results in a failure to discriminate. 

In brder for assessments to be fair and valid, the standards used by assessors 
should be defined expllclty and in as much detail as possible. Where several levels 
or categories of competence are involved, experts in the subject-matter area 
should establish conrespondtng standards before an assessment is made. Ideally, 
there would be a consensus among experts as to the number of categories or 
levels and the standards which define those levels. For the assessment of com- 
mon types of leaming experiences, it may be possible to have predefined levels 
^d standards. For the assessment of more unusual or new leaming experiences, 
it might be necessary to set down objective standards on an ad hoc basis. A 
three-stage process is sugg'ested. The expert wOuld begin with a set of common 
standards written in terms that cut across all areas of leaming and consequently 
are very general. These initial definitions could relate to the assessment framework 
of the institution and its philosophical orientation. In the second stage the expert, or 
experts, would translate these.gen^ral definitions or standards into more specific 



Figure 1 

Genera/ and Specific Definitions for Standards of Wrjting 
Contpetence 



Gertera! Definitions 



Specific Definitions 



Competence is unquestionably well 
above minimal requirements for cred- 
it. 



Writing is clear and precise with excel- 
lent vocabulary and grammatical us- 
ag«rrHasily adapts language and style 
for different purposes. 



Competence is minimally sufficient for 
credit. 



Writing is adequate for freshman 
level, though vocabulary is somewhat 
limited and minor errors in grammar 
are sometimes .n^ade. 



Competence is slightly below minimal 
requirements for credit. 



Writing is sornetimes unclear with 
words occas'ionally used inappro- 
priately. Major ^grammatical errors are 
occasionally made. 



Competence is far be )w minimal re- 
\quiremenls for credit. 



Writing often is incoherent and dis- 
jointed. Vocabulary is extremely lim- 
ited, and major grammatical errors 
are often made. 



and objective definitions related to the assessment area. The final stage would 
involve selecting objective indices or behaviors that relate to the experts' cefini- 
tions of each level. If several experts are involved m setting standards, the objec- 
tive indices or behaviors can be first generated independently by each expert and 
then compared before a consensus is reached. Figure 1 presents an example of 
the first two steps m this suggested process foV the area of writiri^ competence. 
The definitions on the left are very general and could be adapted to fit a particular 
assessment framework. The corresponding specific definitions on the right trans- 
late the general definitions into more Objective terms related to writing compe- 
tence. 

For ri'latively common types of learning, such as writing competence! this 
aefinitional process would be necessary only once. After good standards have 
been established, they can be applied repeatedly to similar assessment problems. 
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'Jn many instances, however, the learning claimed will be of a highly unique nature 
and consequently standards vyill have to be established on an ad hoc basis. The 
same steps can .be performed, even if only one expert judge is involved. Making 
standards explicit wiji, In fact, sen/e as a highly useful framework for structuring the 
aissossment procedure Itself. Weil-defined standards serve as the comerstone for 
valid and reliable assessments, and in cases where dn^ssessmentprocedureJs 
^not predeitermin ed or fixed, can serve as guidelines for deveIoplng.and conducting 
""ffierassessment procedure. There is, in addition, a further benefit to be derived 
frotn this process. Fronn a student's point of view, feedback given in a framework of 
highly: specific standards will be much more useful than feedback given in a very 
general, framework. Thus, the educative benefit of assessment will be enhanced as 
a direct outcome of a process designed to yield more consistent and valid judg- 
ments atx)ut learning. ' . > 

THE ISSUE OF VALIDITY 

In general temis,\he validity of an instrument is regarded as the extent to which a 
process or proceo^re is measuring what it purports to measure. In terms of as- 
sessment, validity qan be thought of as the extent to which the assessment really 
njeasures the student's true learning experience as related to the criteria iestab- 
lished. ResearcherSjin the fields of education and psychology havr distinguished 
five different approaches to evaluating the validity of an assessment procedure. 

Predictive validity refers to the accuracy with which assessments can be used 
to predict later performance in a related area. Predictive validation is of fairly 
limited usefulness in the context pf experiential teaming. There may be Situations, 
however, where an In'stitutlon may wish to evaluate an assessment procedure, in 
part, by how well the assessments agree with later performance in a closely allied 
field. For example, an assessment procedure devised for awarding credit for a 
prerequisite course could be evaluated by companng the assessments made with 
measures of actual performance in the later course. v 

A Second method is called concurrent validity, and refers to a procedure 
whereby a set of assessments is compared with a set of immediately available 
independent measures of the same learning. For example, an assessment proce- 
dure could be evaluated on an experimental basis by comparing the average 
assessment made for a group of students with a known and well-documented lev§l 
of competence in some specific area with the average assessment made for 
another group without such expertise. Such an approach could be used to evaluate 
the usefulness of a performance test designed to m9asure laboratory skills in 
chemistry The average level of performance, as judged by experts, could be 
compared for graduate students in chemistry and undergraduates without exten- 
sive training. A large difference between the two groups in average performance 
would "validate" the assessment procedure. 

Both predictive and concurrent validations are limited in usefulness because 
of the requirement that groups of students be assessed for similar sets of compe- 
tences. The more individualized the assessment for each student, the more difficult 
it becomes ta apply these empirical methods. 

A third approach to the validation of assessment procedures is referred to as 
construct validity The construct validity of an assessment procedure can be 




thbught'Of as all the accumulated suppcrting evidence for the usefulness of that 
procedure. Construct validiiyis a long-range process that evaluates the theoretical 
underpfnhings of an assessnent device or procedure by examining the relation- 
ships between the assessments and other measures which are thoughl to mea- 
.^irej the; same or similar sets of factors., Construct validation is a conceptual 
process used pritfiarily to evaluate instruments that purport to measure traits such 
as^ihtellisence or personality factors. The program of research implied by the 
concept of construct validation would probably apply in an active sense to only a 
very limited numt>er of institutions with very special capabilities and interests. 
Construct validation comes into consideration in a different way for a larger number 
of institutions. In choosing an assessment procedure, or in setting up an assess- 
ment program, the construct validity of various techniques and procedures used 
would be an important consideration. Ror instance, in deciding whether or not the 
ieaderiess> group discussion techhic;ue should be used to assess managerial skills, 
therayailable evidence relating to the construct validity of the leaderless group 
discussion for that purpose should be examined. ^ 

Content validity is the fourth approach to validation and the most useful in the 
present context. As a process, content validity involves a systematic examination 
. of the assessment procedure to determine whether it is designed to elicit behaviors 
or indices that are relevant to the explicit standards established. Content validation 
is almost always a judgmental process and should, if possible, be done by some- 
one other than the expert responsible for devising the assessment procedure. 
Ideally, several experts would evaluate the procodure from the point of view of 
representativeness and comprehensiveness. One approach to the content valida- 
tion of an assessment procedure that may be helpful is first to develop a list of the 
specific objectives of an assessment. The content validity of a proposed assess- 
ment procedure^can then be evaluated by considering whether or not each of the 
listed objectives is met. ^ 

A more elaborate version of the same idea would involve also breaking up the 
assessment procedure into modular components. 7J}q assessment of writing com- 
petence, for example, might be assessed through three separate documents: (1 ) a 
log in which the student.has recorded leaming.experiences resulting from a spon- 
sored work program, (2) an essay treating a general topic broadly related to the 
work experience, and (3) a report dealing with the history of an industry or profes- 
sion closely related to the work experience. 

A potential list of assessment objectives might include the evalulation of the 
following specific areas: 

• vocabulary 

• grammar' 

' • ability to compare and contr^ast 

• narration 

• dialogue 

• use of footr^otes 

A two-way configuration, as shown in Figure 2, could then be used to evaluate the 
content validity of the entire procedure. The entry in each cell could be a rating, 
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reflecting how well a given assessment component covers a given assessment 
o^ebtive, or simply a check indicating whether or not a particular component 
(X)yers a certain objective.' Use of this more elalwrate procedure could be helpful in 
two ways. Rrst, breaking up l^e assessment procedure focuses attention on the 
actual cqgtont of assessment and helps evaluators to avoid making unwarranted 
assurnptions or inferences. Second, the ratings (or checks) can be summed both 
vertically and horizontally. The vertical sums Indicate how well each objective is 
covered by the total assessment procedure, and the honzonta! sums indicate the 
overall usefulness of each assessflient component. Such mformatlon would be 
particularly helpful in revising an assessrtient procedure. If certain critical objec* ^ 

, tives were not being met, the asr>essmont procedure could be extended to include 
measurement of those objectives. On the oiher hand, some components of the 
current assessment, propeduro might be shown as adding very little to the mea- 
surement of objectives and could thus .be eliminated. 

The example given in Figure 2 shows that competence in writing dialogue is 
noi covered by any of the three methods. At this poipt a decision might be made to 
include a; fourth writing sample, such as a short story, which would allow the 
student to demonstrate competence at writing dialogue. Breaking down the as- 
sessment process In this way illustrates how the method of content validation can 
be used to evaluate and revise an assessment procedure before it is even used. If 
certain critteal objectives were not being covered by the assessment procedure, it 
could be changed or extended. 

There is one additional type of validity often discussed by measurement ex- 

^perts^ this is /ace validity, or the extent to which (He assessment procedure ap- 
pears, on its face, to be measuring what it is supposed to measure. For example, 
multiple-choice tests have been used to measure writing competence with consid- 
erable success. The face validity of a multiple-choice test is not especially good, 
(]owever, because the task of choosing mult.ple-ciioice options does not appear to/ 
be as highly related to writing competence as, say, writing an essay: In this 
the essay .voulo have much better face validity than the multiple-choice test. As a 
general rule-of-thUmb for applied measurement, face validity is always desirable. 
Assessment procedures lacking face validity will inevitably be questioned by stu- 
dents, especially those who do poorly Assessments with high face validity tend to 
create a more positive atmosphere, which may, in fact, result in more valid as- 
sessments. 

Any assessment procedure Is only a sample of what a student has ac- 
complished or can accomplish. In order to ensure validity, it is the task of the 
assessor to.make this sample as relevant and as representative as possible In 
some cases the assessor may be limited as jo degree of control over the sample of 
performance or behavior. A product may be presented, for example, as an indica- 
tion of a student^s competence in some artistic or technical area. Even in this type 
of case, however, assessment can be made more valid by taking into account the 
context in which the product was created. Such assessmertts often use interviews 
as a way of gaming information about the context, and It is here that the assessor 
can lend some structure to the process to ensure representativeness and rele- 
^nce. 




Figure 2 

Example of a Two-way Taiile Used to Analyze Content Valtulty 
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Dialogue 



Footnotes 



Overall 
Usefulness 

of Ead'i 
Component 



Log 


2 


2 


1 


3 


0" 


0 


8 


Essay * 




' 3 


3 

\-| 


0 


0 


0 




Report 




3 ' 


' 2 


0 


0 


3 


11 


Adequacy of Content 
for Each Objective 


8 


8 


6 


/ 

3 


0 


3 





Rating Scale: * 

3 Should provide Information highly related to assessment co|0Ctivo 
2 Should provide some information related to assessment objectivi^. 
i Should provide very limited Information related to asseosment ob}ective. 
0 Should provide no information related to assessment objective. 
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REU ABILITY OF ASSESSMENT 

If validity describes how well an assessment procedure measures what it is sup- 
posed to, measure, then reliability can be said to describe how consistently an 
assessment procedure measures what it is measuring. 

Intuitively> the notion that judgments should be made on a consistent and 
reliable basis seems necessary fropp the point of view of fairness to the student. If 
the same iridividual is given widely varying assessments depending upon the 
judge or the assessment technique \used, one would surely question the basic 
equity of the assessment system, the Implications that reliability of judgment has 
for the validity of judgrnent may rx}t be as obvious, however. As a general rule, 
judgments can b^ consistent without being valid, but judgments can never be valid 
without being consistent. Reliability is, in other words, a necessary but not suffi* 
cient condition for validity. 

What Factors Affect Reliability of Assessments? 

It Is important to recognize three major components in the assessment process, 
eadi of which can contribute to the unreliability of judgment: the task, the student, 
and the assessor. First, there is the "task*' or assessment situation itself. A good 
illustration of an unreliable task is an oral test with only two or three questions. 
Although j^is unlikely that such a test would be used in practice, the probability of 
getting an^accurate measure of competence from sucii a test is extremely low 
since the assessment result would probably depend heavily on chance factors 
related to whether or not the questions happen to hit areas witft which the student 
is familiar.* 

The second major component of the assessment process that can contribute 
to the error in the overall asSjBssrTient is the 5»tudent, or person being assessed. 
Despite their lack of relevance to actual knowledge or competence,^ short-term 
changes in mood, physical fatigue, and shifts in anxiety level are some of the 
factors that could cause students' performance to differ. - ' 

The final component Ts, of course, the assessor. The assessor brings a s^t ot 
idiosyncratic standards, attitudes, and preferences that can lead to errors in as- 
sessment. For ex!amp!e, a rater with extremely harsh^standards will make a dnier- 
ent assessment from a rater with very lenient standards. It is this last component 
that concerns us most directly. From the reseaich that has been done on the 
consistency and accuracy with which assessors make judgments, a variety of 
specific sources of error have been identified. Sonje of the more important sources 
of error are as foilowsr * / ' 

1« Leniency or Harshness Error, Some assessor$ tend to nr)ake judgments that 
ace, on the average, much more favorable or more lenient than judgments made by 
other assessors. Conversely^ other assessors may make judgments that are con- 
sistently more unfavorable than the judgments of other assessors. These errors 
are sometimes referred to as leniency and harshness effects and are analogous to 
the phenomenon that students in traditiond! settirjs encounter m "easy" or "hard" 
graders. 

Example. Over a period of six months two professors both evaluate mdepen- 
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d^t reports submitted by 30 students seeking academic cree^tt for a prior 
learning experience. Professor A grants credit to 27 of the students, but Pro- 
* fessor B grants credit to only 8 bludents. The assessments are Inconsistent 
l)eoause Professor A clearly has more lenient standards than Pr<^fessor B. 

it Errors of Central Tendency* Many.assessors are reluctant to commit them- 
selves one w&y or the other and as a consequence tend to make most ratings near 
the dy^re^ or center of a scale. This type'of error, often referred to as the error of 
central tendency, is partcularly troublesome in situations where some discrimina- 
tk)n among individuais In a group is needed. In the indMdual assessment situation, 
errors^ of central tendency will result In a lack of discrimination and lower reliability 
and validity, 
i- 

ExaFnplB. A panel assessment procedure involving four judges is used to 
eyaiuaie the. competence of five different students at piano. Threw of the 
judges agree perfectly and rate two of the students "below standard/* two 
"cHjtstanding," and one "adequate." The fourth" judge, who is less experi- 
enced, "played it safe" and rated all five students "adequate." In doing so, he 
failed to discriminate among levels of student performance; 

3* Cfalo Effect. In situations where a student is being assessed in several different 
specific areas, a favorable overall impression may result in unjustifiably favorable 
judgments in all areas. This type of error is often referred to as the "halo'* effect 
although the reverse can also occur. That is. an unfavorable overall impression 
can result in unjustifiably unfavorable judgments in specific areas. 

Example. A highly articulate, personable student presents three short stories 
for academic credit Id English. Two professors evaluate (l^e stories. Professor 
A has known the student informally for about a year and considers her highly 
/^intelligent. Professor B has never met the student. Professor A judges the 
^ stories to be of acceptable quality for credit, but Projessor B does not. ,Qne 
possible explanation fof the disagreement is that the halo effect caused Pro- 
fessor A to rate thejstories too high. 

4. inltifii Impressions. An 3rror similar to the halo effect can result from the initial 
impression an assessor nas of a student. In some situations it has been shown th^t 
a favorable or unfavorable initial impression yvtll unjustifiably affect later judgments 
of specific aspects of performance. 

Example. A student -about to be interviewed about his travel experiences 
ignores the assessor's outstretched hand and takes a seat. The interview is 
conducted, and the assessor decides not to award credit. The unfavorable 
initial impression made by the student may have caused the assessor to adopt 
harder standards than'woutd usually be the case. 

5. Stereotypes. Strongly neld attitudes or beliefs can cause misperception and 
error in judgment. A good example Is the judge who is influenced by a stereotype of 
members of a particular class or group. It should be pointed out that such errors 
may be .favorable or unfavorable to the student depending on the kind of 
stereotype held. 
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Example, A 65-year-old woman and a 25-year-old nan both submit reports 
based on similar work expen^nce. Though iho reports are equivalent In qual- 
ity, the man Is awarded more credit. The difference m judgments might have 
been caused by storeotypos neld regarding older women. 

6. Contrast Effect The quality of the student who was rated previously will often 
affect judgment. An average studer^t may tend to receive lower than average 
ratings if the previous student was oi^tstanding and higher than average ratings if 
the previous student was poor. ThiSjtype of error has been referred to as the 
contrast effect. 

Example, An assessor intervicv.ctwo students successively regarding similar 
work experiences. The first studen! gives extremely evasive, vague answers 
about the experience and seems generally unable to articulate any learning 
outcomes. The second student describes the experience and states learning 
adequately. The first student is judged to be "well below standard," but the 
second student Is judged "outstanding even though on an absolute basis the 
interview performance was only adequate. 

7. Similarity of Background. The degree of similarity between a judge and the 
person being assessed with respect to background, attitudes, and ethnic group 
has been shown to affect judgments, with greater similarity tending to produce 
more favorable judgments. 

Example, Two students, one black and one white, are both interviev/ed by 
one black and one white assessor. The black assessor rates the black student 
"outstanding' and the white student average. The white assessor ratec the 
black student average" and the white student outstanding." The discrep- 
ancy might be explained by the similarity or difference between assessors and 
students. 

In addition to these specific classes of errors, other factors can affect the 
consistency and accuracy of judgment. It has been shown empincally and dem- 
onstrated theoretically, for example, that the average of several assessors' ratings 
IS more reliable than the ratings of one assessor. This seems to mak& sense 
intuitively when the various sources of error descnt>ed above are considered. Most 
of these are idiosyncratic or unique for a given assessor The advantage of having 
several assessors or judges ts that many of these errors will cancel each other out 
and thus produce more accurate or reliable ratings. 

Ariolher source of error may be found in the transition from observation to 
judgment, particularly if no systematic recording system has been used. Some 
studies have shown, for example, that an assescor s ability to recall specific infor 
mation from an unstructured interview is often quite poor. Judgments may as a 
consequence t>e based on erroneous or irrelevant information. 



Reducing Error 

Error in an assessment process can never be completely eliminated, but research 
has shown that errors can be reduced. Most of the errors descnbed atx)ve become 
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law serious as crilerfon standards become more explicit and objectively defined. 
The expKcft definitions help to focus the assessor's attention on the specific com- 
petence being assessed and help to reduce conjecture and inference that may be 
based on in-elevantj factors. A^techniquelhat can go a long way toward improving 
.assessment, if used correctly/is to provide assessors with devices to aid observa- 
tion, such as rating scales or checklists. Rating scales, if properly constructed, can 
serve both as a translation of criterion standards into objective behavioral terms 
arid as a systematic aid in recording observations. A rating scale will only be 
.worthwhile, however, to the extent that it helps clarify the criterion standards and 
focuses the attention of the assessor on specific, relevant evidence* 

Figure 3 

Example of an Overly General Rating Scale 



OUTSTANDING 



AVERAGE 



POOR 



-9 
-8 
-7 
-6 
-5 
-4 
-3 
-2 
-1 



A very general rating scale such as that presented in Figure 3 will clearly not 
accomplish these purposes. The terms outstanding, average, and poor' are 
too general and unrelated to criterion standards to be of much help. A more useful 
approach is presented in Figure 4. Here the points of the SLcale. particularly the 
critical points, ^have been anchored by clear verbal descriptions related to the 
criterion standards described earlier in Figure 1 . In addition. b*^h9 .x)ral examples 
help to define the scale more fully. Rating scales of this type are referred to as 
behavioiBlly-anchored rating scales and can be developed in situatioris where 
criterion standards are explicitly defined.^ The general definitions on the nght hand 
side of this scale should follow directly from the established critenon standards. 
Specific behaviors should be generated and scaled by experts in the te\Qvan\ 
subject-matter area. Behaviorally anchored rating scales can be quite useful 
where relatively complex performance is being observed, as in a group discussion. 



'A gO(X}()«9crip<ono( how to construct b«ha#io^iV«rxixv ad ^a! 3ca>rsc^Aoe 1^ C SmiiTmnOi,. 
*netraAsUt»on of Exp«<Utions.' pp t49-tSS 
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Flgyro 4 

A BiSthiiVlomHy^nchomt Sca/o for Writing Comp^tcnM 



1 >-*3k^* '.w 



t<^*9 . fn.ass<jss«nQ a proOi^i? &uc^ iho ve't^^ <}0sct'pt?o<vs may bo a*i lifiaJ 
$5 leasifc<o Of oeoos^^ry Tpm? eHort -nvo^t^' -r v1ovt3iv?p<o9 good t&ung scales jS 

In highly amq^^e o^^MSSsjmoni t>iruai»**^.&i 4 not t>Q jf♦^^lt^»3b^Q so ^»*«k)p 
•^duai rattng ^ta^. ooi SN? di&Jinii#on& itai gofrespojxj lo iNj ai!Gf»on siandajfd*» 
$hOMidi iHi o«pSi*.*ry consideicKJ i^ti^r, an dd^se^Si^ manes <4 iwd^mont T>i0 caso 

inslilul?c^5 ot ago^m must contw d periiXlca^Jy Ai^!^ov9^w stieps war. be laKer ic 
tfto OftOf «n Afjy assessment p;0C6<Jufe tfni> wannoi b^ dono w<1bout some 
cost iho case a* unique asse?i5^monis, i^o oxpcm&o o? It .ijiop<n9 a jatn^ 
sysiefn and a slr\^?ed asjtessmeot p/oc6<Ji/fO may be pr<^b*t»cr 

mafief rx>w catetaihf dove»ope<J a tai«n9 sca>o s howo w©« . it *s ^poriam !c 
rooo<|n?£s ♦mponana? preparing <x ^annifjg asses$ofs S<nc« mai^/ of stie 
prob^f^ ^-1^ jud9 menial »^s*&e5»^imeft^ t40«- n<j»v«dud» a&&05'^>Oft» opefai'>9 
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bo to ensuTD !hal assessors are operating ivithin a con..ron frameworx ol 
and critofia. 

EVALUATION OF ASSESSMENT 

The perkxfic evaluation of assessment should be a permanent feature of an> 
assessment program, biit evaluation of assessment procedures should aiso be 
made whenever a r>ew procedure is developed or revised in ail cases a staff 
memboj or consuttani with some expenSse educabona* and psychoiogicai mea 
surement shoufd monitoc the evaluation, Jf this .s not pos&.o<e, booKs such as thpse 
referoTKod at ttie end of thus chapter should be cor^sulted. Followng ara examples 
of some appcoaches that might be tanen ;o the ostimatiu.. of feDat)ilii> and validity 
of assessment. 




Assessing Reliability of Assessment 

1. RellabltHy of Overall Aaseasment Procedure, As stated eaitei. lOcons^steocy. 
or error in the assessment f>rocess can anse tron. »he assessor, the student, or the 
assessment task or s^tuaUon Before attempting to separate these separate 
souces of error, an evaluation should bo made of the reiial)»Irty of . ;<e overall 
assessment procedure If rebabefity s adequate. 4 -nay not be necessar> to investi- 
gate sources of error 

The key to relrabilitv estimation .s <n obtaining two or nwre independent as- 
sessments for a san^ple of students Tw be re^abie the extent of agreement among 
<ndependertt assessments shouid be high Ootammg -noepenoe. : assessments 
might be relalivefy easy for products oi wrmen materia*. Out can te qurte expensive 
for Interviev^s or performance assessment. An tniervtew or performarK:e wouid 
have to bo repeated with a Afferent expert oi set of experts to ootain indeperKleni 
judgments Although difficult. 4 <s usiialfy poss<bie to carry out such procedures on 
en experimental bas.s A r andonr. sample J students oouid be mterviewed twsca by 
different assessors, for exarr^pie M s important to Keep m mind that the enure 
assessment procedure must be repeated independently »n order to ot3tain an accw- 
rate estimate of asv*ssnnent io^^abihty A product, foi example, might t>e assessed 
Ijy first <nterv*0wrng a student for Iwcxground rfonnatton and then making a |udg 
ment Both the ^nlervfew and the judgment should be done indeperxJentry Oy a 
second expert il reliability of the entire proc^ure »s to be estimated 



2, Ftellablllty of Judges. Ir jfdet Sc estriale ihe retiabiirry of ead^. potentJ/^^ source 
of error, it is necessary to ho<d the other sources constant .n 4^me way Thus, the 
reliat>riity of judges car. be assessed ooiy A the sources of error due to students and 
the task can be controlled Re&a txfety of |u« /gments obtained from interviews can be 
estimated by having ^vc or nxjre ,udges present donng the same interview, or by 
having different judges mane assessments of the same sei of taped or videotaped 
interviews Similar procedures couid be usod for performance assea:>ment For 
products af>d written materta.. a se: o* sample mater^ats couid be evaluated by 
tSffereni judges v. are ^nlerv.ews are employed as an adjunct, transcnpu or 
recordings of the mter* ews could be made available to the fudges 
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3- T«tk Rtllablllty. In situations where a structured task is involved, it is often 
possible to estimate the reiia;)iRty of the task, apart from other sources of error 
With some assessiront proceov^esjt may t>e possit)le to s^araie a given perlor- 
roance.Jnterview, or emulation task mto ^^parallel" parts--parts of equal difficulty 
thalmeasure approximately the same things. A high degree of agreement between 
scores or judgments formed on the basis of perfomiance of the two parallel parts 
would indicate good internal consistency of the assessment procedure This might 
be done, for example, with a performance test consisting of two separately timed 
parallef parts? 

4, RaHability of Student Performance. Esumalmg reliability of student perfor- 
manco fe. pertiaps the most difftcutt source of error to isolate since it requires 
obtainirig samples of perfomiance, products, or written material on two or more 
separate occastons. It may be quite difficult to keep the different assessments 
independent in the sense required. Students may repeal aspects of a performance 
pr answers to a senes of questjori'- because they remember their behavior on the 
previous occasions, or it may simply be inappropriate to repeal the same Inten/lew 
or performance assessment., Because of this type of problem. K is often r;ecessary 
to r,esort to parallel tasks and'to design a study that separates error due to student 
unreliatMlity from error due to task unreliability A problem is also encountered with 
products^and wnlten malenal, particularly where one unique item is being asses-" 
sedJt may not be possible for the student to 'repear the performance, in any inje 
sense. Although assessment errpr due to student unreliability is often difficult to 
assess, ft is also probably less troublesome than judge or task reliability 

5. Separating Sources of Error Statistically- in special cases it may be possible 
to separate sources of tipreliability and identify the magnitude of error In each 
source. Nonnaily- this would require a very carefully planned experiment In which 
students, judges, and subtasks are sampled on a systen>atic basis A statistical 
procedure, known as analysis of vanance, can then be used to break up and 
isolate the sources of error m assessment. An expert with a thorough knowledge of 
expenmenial design and statistics should be involved in the planning of such an 
experiment. 

Assessing the Validity of Assessment 

Evaluating the validity of an assessment procedure also requires the establish- 
ment of special expenmenls or data collection procedures In one sense, validation 
can be thdbgm of as a venfication process m that the results of assessment are 
venfied with information relating to the competence or knowledge measured 
Follov/ing are some examples of approaches to validating assessment proce- 
dures. 

1. Predictive Validity, in cases where credit is assigned for a prerequisite, it may 
be possible to design an expenmenl where students would be rated as to the^^ 
readiness for taking an advanced course by a given assessment procedure For 
expenmenial purposes a sample of students could be allowed to enter the ad- 
vanced course regardless of the rating they obtained jn the assessment The 
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:assessment ratings could then be correlated with advanced course performance. 



2..^hcurrtht Validity. It woujd be possible to choose a sample of students 
ranging, considerably In documented leaming experiences in a qiven area. An 
iassqssmefitprpcedure could then be used to modsure the leaming or competence 
of each student without making this evidence availabie to the judges. The results of 
the assessments could then be compared with the documentecJ evidence. This 
method might be appropriate for an oral test or a performance test and would be 
particularly appropriate for product assessment. It would clearly not be appropriate 
for ap Interview, when one of the functions of the interview is obtaining information 
about leaming experiences. 

3/Content Validity. An empirical approach to content validity can be taken by 
having two independent teams of experts devise an assessment procedure given 
the same set of initial criterion standards. A sample of students could then be 
assessed with each procedure, and the overall assessments obtr'ned with each 
procedure could be related. A high relationship would indicate content validity for 
both procedures. 

4. Special Investigations of Accuracy. For some procedures, such as inter- 
views, it may nprmally be very expensive to check into the accuracy of all the 
information obtained from a student during an assessment interview. However, it 
would be possible on an experimental basis to use extensive checking procedures 
and thereby obtain an estimate of the accuracy of information obtained during the 
interview. 



Judgmental Assessment of Assessment ^ 

In order for assessment to be judged without conducting an aclua; expenmenlal 
study, it is first necessary that a very careful documentation be made of the entire 
assessrtienl process Whether the procedure is a highly structured performance 
test or an unstructured Interview, this documentation means that critenon stan- 
dards should be written out. a plan for evaluatipn should be explicitly written, and 
*-ow the plan for evaJualion relates to the critenon standards should be made 
explicit For interviews and performance assessment, electronic recording plus 
'^me written rec rd of observation should be made. Finally, the assessor should 
state how the decision or final overall judgment relate^ to the observations and 
criterion standards Given such a systematic reqord of an assessme;it procedure, it 
is possible to have several experts judgmenlally assess the internal consistency of 
the assessment and the relevance of the final judgment. The experts assessing the 
assessment should include at least one expert in the content area anu at least one 
expert in measurement. 

RECOMMENDED BASIC STEPS FOR CONDUCTING ASSESSMENTS 

What follows is an attempt to summarize in a genera! way a set of b^sic steps for 
improving assessment applicabie at different levels to any assessment situation. In 
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developing, revving, and.monitoring ar\ assessment procedure, it will be neces- 
sary to involve several different points of view— that of the expert in a specific 
substantive area, that of the expert in the methods and problems of assessment, 
and that of the students who will be assessed. 

f, 1. Establish clear criterion standards. This should be done in explicit, observ- 
able teims that have a clear meaning for both the assessor and the stu- 
dent. 

a. Begin with general definitions of competence or leaming that can be 
applied to all fields. N 
T--^. Define specific learning outcomes that should be assessed, 
•^c. Write definitions that can be specifically applied to the area of leaming 

in ^hich tiie student is seeking credit, 
d. Identify critical behaviors or indices that relate to different levels of 
competence or leaming. This is particularly important when the as- 
sessment .can lead to varying numbers of credits. 

2. Select and structure an assessment procedure that will elicit a sample of 
observable behaviors, indices, or other evidence representative of the 

; criterion standards. ^ 

a. First determine whether the proposed assessment procedure is content 
valid. Use a systematic approaph (such as that illustrated in Figure 2). 

b. Revise and extend the procedure as necessary to ensure that all impor- 
tant objectives are covered. 

3. Plan the administration of assessment. 

» 

a. If more than one assessment task or modular component is used, the 
I order in which these components are administered should be specified. 

b. Make sure that Instructions for both students and assessors are clear. 

c. Make sure that provisions are made for the required space, time, and 
equipment. 

d. Choose assessors carefully and make sure that they understand the 
procedures fully. 

e. Provide for recording of observations electronically, through rating 
scales,'bhecklists. or detailed note-taking. 

f. If possible, try out the assessment method with one or more students. 
The tryout could involve an actual student or could utilize role-playing 
techniques. 

g. Based on the tryout, revise and make final procedures for administra- 
tion. 

4. Ensure that judgments are based on what has been observed. 

a. Try to have at least two assessors. 

b. Familianze assessors w;th common types of errors made in assess- 
ment. 

c. Provide assessors with observational aids such as rating scales or 
checklists. 
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Vd.*^ Establish the requirement that all assessors describe in a brief written 
report the observations they made and how those observations relate to 
their assessment of the student's competence or learning. 

5, Record and report the results of assessment. 

a. Scores should be reported in^ermanent records, Including student rec- 
ords and those designed for data collection, research, and reporting 

br Set up procedures for reporting and interpreting test results to students, 
which' would include; 

(1) a written report to the student containing score Interpretations. 

(2) an interview covering interpretation, diagnosis of leaming problems, 
and suggestions for further study or practice on performance' skills. 

6. Monitor assessment procedures. 

a. Detailed records and reports can serve as fnp Jts for evaluating the 
effectiveness of assessment. Someone with methodological experience 
and expertise in assessment should periodically monitor such records . 
to determine whether problems exist and,, if so. what can be done to 
correct them. 

b. A second monitoring procedure .would involve having a methodological 
expert directly observe assessments. Direct\observation may suggest 
problems not discernible through written reports. . 

c. A more elaborate procedure for monitoring assessment would involve 
conducting empirical studies of reliability and' validity, such as those 
suggested in the preceding section. 
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■ The Interview and Related Procedur^es 



Uudith Pendefgrass, Jane Porter Stutz, and Richard Reilly 

Of the four specific techniques discussed in this Handbook, the interview and its 
variants are perhaps the most commonly used for the assessment of experiential 
learning. 

When an interview or. for that matter. a>iy measurement technique is part of 
the assessment process, its exact functions depend upon the institution where it Is 
to be used and can be considered only in the context of Institutional goals. Thus, 
the purpose of an assessment im^iew at one college may be quite different from 
the purpose of a s^imilar interview al another institution. Similarly, the suggestions 
offered later In this chapter for conducting an Interview or other assessment will be 
applicable at some instituUons and not others, and within a single institution in 
some situations and not others. It is not the Intention of this chapter to dictate what 
the purpose of an assessment technique should be, but rather to describe some 
functions that suci) assessment might serve and to provide some common-sense 
hints that may prove useful in assessment. It must be left to an Individual institution 
to determine what is appropriate in its particufar'case, 

FUNCTIONS OF THE INTERVIEW 

Some functions of Interviews that relate to but do not actually constitute dbsess- 
ment are 

• obtaining information 

• communicating information 

• influencing or changing behavior 

• diagnosing problems 

• verifying documentation 



For example, the exploratory interview Is often used at institutic. that credit 
experiential learning to gather information about a student's background, interests, 
goals, etc . from which optjons can be suggested that the student may follow. In 
this type of situation the interviewer may be said in some vagu , senf^ to be 
"assessing" the student, but such an interview does not constitute assesbment in 
the s6nse of measuring a specific learning outcome in terms of what it is worth 
toward a college degree The exploratory interview can be considered one type of 
"counseling inten/iew," and intervievys of this sort— in which progress is discussed 
and options suggested— may be employed throughout the student's college 
career Counseling Interviews may also be used to prepare a student for a panel 
interview or other assessment or In some way to influence a student s behavior. 

A very common use of the interview is 'erificatlon of documentation, which is 
not assessment In the strict sense. If, for example, a student brings in a letter of 
reference, the inten/iewer usually asks some questions about the experience refer- 
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red to In the letter. In this case, there is no product or performance to be assessed 
directly.vthere Is merely description. 

' interviewing for diagnostic purposes is also common in education. Ono exam- 
ple is the holistic interview conducted io determine wl ether theje are gaps in a 
student's. education, A counseling inten/iew that results in recommendations for 
next steps Is also, in a sense, diagnostic. 

« Iri tenns of assessment through oral procedures— the focus of this chapter— 
the Interview functions suggested so far are secondary. The primary function of an 
assessnpent interview, is to obtain Information or evidence so that a well-informed 
judgment can be made regarding a student's competence or knowledge. An oral 
procedure may be used to assess knowledge or skills directly, as in an oral e)&mi- 
,*nation; or to supplement another means of assessment, as in the case in which an 
inten/iew provides background and context for assessment of a product such as a 
short story or a musical composition. \ . 

Mliny of the hundreds of research studies that have been done over the years 
have concluded that inten/iews, particularly unstructured inten/iews, lack reliability 
and validity, it is encouraging to note, therefore, that the GAEL validation^ study of 
assessrpen) interviews reported both reliability jand validity at acceptable levels. In 
brief, this siudy found that there wassi»bstantiaLagreement (reliability) between an 
interviewer*s and an observer's judgments of an inten/iewee's counseling compe- 
tence. Moreover, there was a significant relationship (validity) between assess- 
ments fnade with the interview and independent assessments made with a role- 
playing technique. These results provide perhaps the first systematic research 
v^vidonce to support the use of interviews to assess experiential learning. " 

TYPES OF ASSESSMENT INTERVIEWS^ 

An inten/iew can be defined as an oral exchange between two or more people that 
differs from an ordinary conversation in that there is some structure, there is a goal 
or purpose, and the roles of the participants are'dearly defined. Four more or less 
distinct types of assessment interviews can be identified: the f)ral examination, the 
one-to-one. interview, the panel interview, and the leaderleis group discussion, 
within each of which a range of structure is possible. 

In each case, a detailed schedule may be drawn up for conducting the as- 
sessment, or a list of discussion topics may be used. Many institutions Citillzing. 
these techniques employ the whole range, determining the amount of structure 
necessary on the basis of the knowledge to be measured, the evaluator's wishes, 
and the skills of the individual student. Consequently, variation in specificity of 
^ detail exists among the four types of Interview techniques as well as in the ways 
.^th^t institutions design them for the evaluation of a particula: competence. 

Oral Examination. An oral examination is an interview, usually highly structured, 
in which questions are planned in advance and relate directly to the competence 
being assessed. Often preset standards deterrnine the assessment decision re- 



'S«» Warren W. Wittngham and Associat&s. The CAEL Vahdabon Report. Chapter IV. 

•For »r»othef(i5cu$sonot types of intOfvwwsosed'w asiQ^itner^.s^ A Coniendium of Assessment Teoi fvques by Joan 
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Ja^cling::the student's competence. Oral examinations are used in fields like 
^m6<^ to determine certification and licensing, and in fields JIke counseling, 
Where inteiperspn competence is assessed; The oral examination is a relatively 
.,objective:demo competence or knowledge. It provides an opportunity 

for tKe exairiner(^^ for clarification or justification, or to judge the manner 
and cpnten) of the reply to a direct question, and it has great potential for use In 
cpnjunctior{ with, performance testing. The major disadvantage of the oral exam is 
that It Is time-consuming and costly (compared to a group paper-and-pencii e^an»). 

.One-tQ-boe Interview. VVhea one interviewer questions one student, tho inter* 
viewer c£Ui devote full attention to a single individual and can. in some cases, adapt 
the questk^nlng (style, technique; and content) accordingly. There is a range of 
irtrfacture In one-to-one inten/igws. A very structured inten/iew. makes use of a 
^''scheduiy'-l-a list of specifiaquestions to be asked In a predetermined order. 1n a 
semisfructured^ inten/iew, thejnten/iewer knows exactly what information is re- 
quired but Varies the wording and sequence of questions for maximum effective- 
ness with Individual, students.^ In the least structured one-to-one inten/iew, the 
inten/fewer determines in advance the areas or topics to be covered but varies the* 
questions with the specific situation. Since inten/lews are depen'dent to some 
exter\t upon personality, they are less reliable than other forms of assessment; 
obviously. structured Interviews are more reliable than unstructured interviews. An 
advantage of the one-to-one interview is that it may require relatively little prepara- 
tion compared with other procedures. The disadvantages are, again, the cost and 
thei'me required. Many experiential learning situations, ranging from foreign travel 
to volunteer community work, lend themselves to assessment by one-to-one inter- 
views. ^ 

Panel. interview. In a panel inten/iew, several inten/iewers^questlon a single stu- 
dent during one session. This is, of course, less tin[ie-consuming for the inter- 
viewee, than being interviewed by each examiner separajely, and the inten/iewers 
can benefit collectively from each other's questions and group discussioh. The 
, panel inten/iew can also sen/e as a simulation of a situation where the individual 
must account to a group. On the other hand, tHe panel inten/iew is subject to 
"contamination"; i.e., inten/iewers' comments an;) questions may influence each 
other's views and lead them to a different evaluation from one that they would have 
arrived at individually. The panel members also need to take care that the inten/iew 
does not become a "trial by jury." Finally, the parjel inten/iew may be especially 
anxiety- producing for the student; although this maybe an advantage if the compe- 
tence \o be assessed involves ability to^act or think under stress, normally students 
can be assessed best if they are at eas^ \ 

Leaderless Group Discussion. In this type of oral assessment several 
examinees are asked as a group to carry on a discussion while the examiner(s) 
obsen/e. No leader-Is appointed, and the examiners do not enter the discussion. 
^ The leaderless group discussion (LGD) is often us^d to assess competences 
related to leadership skills. As with other oral techritques. there is a range of 
structure possible within LDGs. At one extreme, a very general topic may be 
presented for discussion, at the other extreme, each participant may be given 
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detailed background information and asked to play a specific role. The advantages 
of the LGD are that it can elicit a type of interpersonal behavior with peers that is 
not readlly.observable In other formats, and tt can provide a direct learning experi- 
ence and feedback for the student. The disadvantages are that the 'amount of 
strui;ture that can be .imposed is limited, and as a result the validity and reliability 
are also limited. Furthermore, several assessors are needed. 

DEVELOPiNG THE INTERVIEW 
Key Steps ^ 

This section offers, specific recommendations for the development of the interview 
itself.^ In each case, the recommendations are fairly elaborate and assume exten- 
sive resourcesjn terms of personnel and time. It is recognized, however, that 
resources are often quite limited and that it may not always be desirable or feasible 
to carry out all of the steps. Each of the suggested steps will usually have to be 
considered in light of the assessment problem and the existing resources. Carrying 
out all of the steps would be especially desirable for assessment situations that are 
likely to be repeated many times. 

1. Establish Criterion Standards and Determine Appropriate Levels. The first 
two steps in the development of an assessment interview are common to each of 
the specific procedures and essentially involve the establishing of criterion stan- 
dards discussed in Chapter II. Thalis. the experts should first determine the levels 
of competence In the specific area to be assessed.by writing fairly objective defini- 
tfons (as In Rgure 1) and then, as a second step, should define those levels in 
objective observable terms. These two steps will then guide the development of 
the complete interview. 

2. Develop the Content. Once criterion standards have been established and 
explicitly defined, the interview schedule itself should be drawn up Questions 
should be designed so that they are directly related to the criterion standards A 
representative sampling of questions should be developed to cover the subject 
matter. Questions may be of varying difficulty and should be arranged^ systemat 
ically, such as moving from the easier to the more difficult ones. 

3. Develop Observational and Recording Procedures. A procedure for record- 
ing and rating students responses must be developed m advance of the adminis- 
tration of the procedure. A rating scale or checklist, with space provided for com 
ments, may prove to be most useful. The ^3e of such a recording instrument is 
especially important in a teaderle^s group discussion. Behaviorally anchored rat 
4ng scales, such as the one presented in Figure 4. could be developed or checklists 
of behaviors could be prepared. In each case, the rating scale should be closely 
tied to the criterion standards. 
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^. Jf)/ouif Evaiuationi and Revision. All questions, materials, and rating scales 
st>puld be.tried out with a small sample of students and more than one judge. This 
will allow;an^lgurties to be cleared up and unnecessary questions dropped. Of 
course, as has been mentioned, an elaborate try-out procedure may be impossi- 
bte. In-general, howeverycsome.attempt should be made to determine if the as- 
"sessment process elidts responses that are adequate as well as relevant to the 
jcriterfon standards. . 

•5. Dtvtiopmtnt of Final Products. A final set of questions or discussion topics, 
guidelines for administration, and a recording form should be produced. In the case 
of a leadarless group discussion, a set of background malenals for students should 
belnciuded; 

Designing The Interview Schedule 

EstabHshing Content and Criterion Standards. Rrst, the persons responsible 
for designing the schedule must determine the categories of knowledge and skill 
^fundamental to thd area of learning to be assessed. In some institutions several , 
consultants in the field may aid in this process, and the responsibility for complet- 
ing the interview schedule itself may be given to a member of a special assess- 
ment staff or to another faculty member. This approach is particularly common in 
cases in which rather structured schedules are being developed or several 
eyaluatprs will be using a particular schedule over a period of time. 

For example. In developing the interview for measuring leadership skill for a 
volunteer organization (included at the end of this chapter), consultants identified 
twelve such categories. Although these categories differ somewhat in scope, ea^d 
Is actually composed of several facets, and it is important that the questions used 
within the interview be designed to cover some but not necessarily all of the 
sut)components in each of the categories. Although there are not required answers 
that are necessary to demonstrate competence in the area, the student should 
possess two abilities that cut across all twelve subjects. These are the ability to 
analyze beha^'iors and the reasons for them and the ability to compare organiza- 
tions and activities in order to determine the generalizabllity of the knowledge 
gained. It is important to note at this point that such an interview is designed not to 
validate the student's participation in^ set of experiences, but rather to assess the 
outcomes of those experiences. 

Following agreement on the basic categories of knowledge and skills, a 
specific set of criterion standards is derived that will later serve as the b&tis of a 
rating scale to be used during the administration of the interview itself. For exam- 
ple, consultants .derived the following standards for leadership competence m. 
volunteer organizations. A student should: 

1. Understand arid apply the goal-setting proces. , understand the organize- 
tion*s goals and evaluate them in setting prionties. 

2. Uri^erstand and apply tjie processes of organizing new organizations and 
maintaining established ones, evaluate similarities and differences be- 
tween the two types of processes. 

3. Understand the structure of the organization and apply this knowledge by 



ERIC 



ne 



33 



analyzing and maintaining the relationship between the structure ai 1 the 
organization's goals and central identity. 

4. Understand the use of a constitution and its effects on the directions taken 
by the organization; understand most provisions of this organization's 
constitution, if any. 

5. Understand own leadership style and skills and apply this knowledge in a 
variety of situations within the organization* arid evaluate outcomes. 

6. Understand the organization's sources of revenue 'and fiscal manage- 
ment processes; apply this knowledge In determining and evaluating al- 
temate sources of revenue and accompanying management techniques, 

7. Understand'the role of various types of communications in performing 
. leadership functions and apply this knowledge in-anal/zing their varia- 

^tions in diiferent organizations; evaluate own use of communications 
skills Jn relation to a specific organization. . 

8. Understand* apply* and evaluate the value and techniques of using public 
communicatioris^media in achieving an organization's goals. 

9. Understand a variety of sources of Information and community resources 
and apply this knowledge m locating and evaluating them in order to 
complete a task. 

10. Understand sources of decision making extemal to the organization and 
apply this understanding by analyzing the relationship established be- 
tween them and thb organization. 

11. Understand the civic arenas in which volunteer organizations may oper- 
ate and apply this knowledge by analyzing the arena In which this organi- 
zation operated and one's own role in being responsive to that arena. 

12. Understand andjapply the techniques of supervising volunteer and paid 



Determining Appropriateness of Interview Technique. The next step is to de- 
cide whether the interview is an appropnate assessment tecnnique for m^sunng 
a particular body of knowledge. This determination should be made in light of the 
criterion standards that have been established! An interview alone may be 
adequate to assess the competence, though in some cashes an additional method, 
such as a product assessment, should t>e used. 

The interview is especially useful in measuring knowledge of leadership 
techniques because problem-solving situations may be posed to determine the 
generalizability of the knowledge. The interview also allows the assessor to 'dcus 
the interview on special areas of competence as they surface dunng the course of 
the interview. Frequently the specific direction cannot be fully anticipated t)efore 
the interviewer and student become acquainted and the interviewer becomes more 
fully informed of the student's experiences. 

Furthermore, the affective dimension of the competence area is a significant 
one to measure since it frequently plays a substantial role tn determining the 
direction that a leader of an organization chooses to take and the result of his or 
her endeavors. The interview is particularly effective in evaluating a student's 
values and attitudes 
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IHwopInfl :th« Qu.stFon Series. Next a spt of questions-the interview 
■s^«kJl©T^ls drawn up tor use In moasup'ng tne khowlec^e and sfdll being claimed 
by the student. Using a somewhat structured schedule aids in achieving consls- 
tency among the Interviewers who will l» assessing the competence. It also pro- 
motes tnore thorough coverage of the subject as well as similar treatment of all 
students cialreing this knowledge. In many cases time and cost prohibit the de- 
•vetopment of a complete schedule. 
- 'In developing the schedule'it is essential to work within the confines dictated 
hk!;® Jtandards. Each category should also be covered thoroughly, 

although it is not necessary to separate questions by categories of knowledge. It 
may sometimes be preferable to design questions that touch on several compo- 
nents at once. This has the advantage of allowing the student to demonstrate 
understancSng of >,he relationships among various facets of the competence 
_ Following the development of the schedule, a .-ating scale should be designed 
that !s based ort the criterion standards as well as.the schedule rtself. An example 
IS Includ^ at the end of this chapter. The scale stibuld be easy to complete and 
useruf mtnaking final credit recommendations. Space shouW bo allowed for com- 
merits. * 

K«y Qutsllqns for Use In Interviewing, The follov^ii^ typos of questions may 
prove particularly appropriate for measuring expedenllal learning, jn general these 
questions emphasize the Importance of the student's articulaUon of the knowledge 
she or he tias gained Though similar questions could well be used for evaluating 
more classic acadefmic areas, adjustmsnis should be made In order to solicit fewer 
personal perspectives and more specific Information. In this case the range of 
acceptable responses would not be as great. Questions may require the student 

1. Recall activities and evaluate their priority aruJ value. 

2 Analyze issues in a given dimension and evaluate his or her ability to deaj 
with them, 

3, Draw comparisons among experiences. 

4 Use hypothetical situations Jo demonstrate problenvsolvmg skill within a 
given context. 

* 

5 Identify liends or systemaUc changes in his or her acDons as well as the 
reason for them. 

. 6 Analyze his or her own strengths and weaknesses with regard to specify 
. situations or sets of behavior- 

7. Analyze his or her basic operating style. 

8. Identify theories that are consciously be^ng applied 

Reviewing for Content Validity. Finnlly. ti mter.-ew schedule should be 
examined for content validity, ideally by the osnsuJtanLs who first served m its 
formaUon as well as by other experts in the area. Tho allows the institution to 
check to be sure that the judgments of the milia! subject-maner experts have been 
maintained It is also beneficial to test the instrument with persons who have fresh 
perspectives In many cases the Interviewer will be the solo sot ^ct-matter expert 
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iiav^ anottw poison oxponorced mo field iev»e* me sdiedute beto'O ss pu« 
fe«o feoqt>ont uss 

CONDUCTING THE INTERVIEW 

•ma first step m prepanng icf ao asses&moni ,nit}fv«*v .-s tne .dontxhcauon o« iho 
assossof.- IrfeaBy. the iweme^vst shaaid have expert Knowledge o' the field as we.i 
8s»»lafViev/fng expewnco Some instnutsons use a par 3 team o» .ni0«vte«G«$ 
my/m\ both sut^'-maiiet expomso and mterviowing sksHs ate represented 0« 
coistm the -idaar situatjon is not usuaKy ai hand, twt the pofson who conducts 
the Bitefview shouKJ m any case be cooperai-ve. tiexfbto. lotetanl of diHer em kinds 
of urtqa».^9tpef!anc©s, amJ wang to use gutdetms (For suggosttons on tdenttty- 
ing peopte w«h sut^ect-maner expertise, see Chapief i ) 

P»tor 10 the traefv m rtserf. it « des!f able tor iho .nterviewer and tho assossee 
to meet es^i olhw, w .f mat .s not poss»b?e, to talk on the teiephono Ttis has« 
potposo ot this conversation to famikams tho assessor wlh (ha sJutfant s srtua- 
l»n and to gwe the stuiJeni an ove*viow oi what can os expected dunng the 
assessment m aOtftion enter «>« standards can be agreed upon a! this and 
m muss to be covered can fca exp5a«noa so inai the studem can f etlod on top«s 
ho Of she wshijs to inciude .n responses This approach often produces mjie 
thofough answers whrfe *eserv.r>9 the elements of fresh rfwes?»ga»an and analysis 
ot KnowtetJge for the .mervrew <ts8H Some .ntefviewers also choose to tSscuss the 
rafing scale This rmv^t^ before tha .merv.ew g.v«s the assessor and the student 
a &mai to become acqu-a- ted. .1 orty bneity and can axvo 10 ««duce the 
suittonrs arwmiy at lU} tm., of the assessment At !n,s !>me the delaiis of nme 
ptaco arKJ kjrvglh of the* inietvuew can mo 00 fmai'Zed 

PfOiniervH)w preparation iot im assessor involves piannmg me mwnflew and 
adiustsng the mten^.ew schedule to ma (mntoMt student s suuauo" For oxamps^ 
.n the case of me leadefsmp competenc* schedule d<sc«ssed mf^>& the pnnc>p3' 
organizalwfs) .^wh«h the sHidont has pan^opaied should be reflected lhfougr> 
out the imefv.mv Th»s rriay f>ecoss.iate me adStJon of some specie terms & 
in'ormaSKjn 10 som« of (he qu6St>ons Tn.s prototype was des^gneO wih iho unde< 
siandioa " s^"«e«t wo.jK3 be asHsd to draw conclusions most neav-iy from 
'■ih* ofgnn«3i<wi 'n when mt '.ne n.id.ia«ef^ a leadersh-p roie. ^ thai 

rtKorootauon of e-«p«twrKx.s from otrter orga^^-^aKons .1 any wore app^bie 
woOKJ bo encouraged The student wouHi also be urged to nvpotr>o!W» rogard.og 
cthOf sstueuor^ fsased on Krwwieo . >* if" '■-no orgamzat-on .n wtt-ch or she 
t^ad partffispated ext8ns»vo^ 

in adfflimstenng a protoiype 'ntfirvtow a ma, bf necessary to oLminaie some 
ttsfnsmararsr^otappropnatetoineoxpo'wncesofthesturtcnt Oh/'OusV there -s 
a w«Jo wa/^oty among studor,!^ wiumng exper .onces ar^o me speo'K ^ .iis denveo 
from those expernsnccs m order to bo rnost useful as an assessm..^ !echn«,ue 
ine interview sc*»sduie mu6t trwn refiec? l^^ p'ocui.ar set o- ^nowieogs and swis 
thattha atudoot brtng-^ 'r, itK. mer^-a* How^a^e' me core c».tor ustabhshedby 
Bio (n^lut-oT' toja^-^ram-^ ■■ 'earn.r»9 of compotc-vc-e wr'h.n im amn should 
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remain constant among vanous students, Ccn&equentFy. the basic interview 
schedule and components of the compeienwe atea shoutd atso remain (Oiauvef^ 
constant 

Some Practical Suggestions 

A number of handbooks and manuals have i)een wnlten on how to intemew. 
partfcular^y In the area of employment ^nterviewniv^. Several of tnem d<e muudeo in 
the bibliography at Che end of this chapter, Th^s section presents those heiptu» 
hk^ls*^ that apply particuJarty to assessment interviews. There is nothing esotenc or 
even technical about these suggestions, mosl of them are meiery common-sense 
Ideas, but awareness of them can improve the QtfecUvenes^s of the ihierview 

Interviews or other oral assessment procedures of any sort are t>es! con- 
A)Ct0d it\ privacy with as few ^ntern^ptions and (istractions as possible « is a good 
fdea to find a room where voters cannot be overheard and to arrange to have no 
phone calls or visitors dunng the course of the assessment Tne interviewer should 
be sure to arrtve on time and come prepared with copte^ of tx)th the interview 
sche{kjfe and the rating scale so that the gr<d can be filled out as soon as the 
inteni^ew is completed \* a pane* Evolved, copies shouid be provided lor each 
member When feasible .t can be heipfu* to conduct pr^tice interviews and to 
complete practice rating forms bclofe conducting a real interview 

The first few mmutg^s of a^- «^lerv»ew oi orai examination are important be- 
cause they set the lone fo? aK shat ioikrHh The student wii^ oe abie to demonstrate 
competence best 4 or she .s at ease, and she .nterviewei can heip by creating as 
relaxed and pleasant ar atmosphere as possiUe This can t>e accomplished oy 
greeting the student pleasant^V and gt^^ctg the distinct .mptessjon shat it will oe easy 
10 t^k with one another Some mdnu*^.s suggest a tew mmutes ot small tarn to 
help reduce ***9 students anxiety The interviewe* should have the fiexjbiUty 
necessary to adjust the schedule to uc he? own styJe as wen as to the student s 
Such adjustment .s necessa> y for majf ^^.n^ng a smooth flow within the mten^iewrng 
process as well as a connlortat>ie s.tuaw . for the student Th^s, in turn, is essential 
tn o^der to max/mize the effectiveness a e*f«uenuy of me assessment technique 
tt IS atso important 20 clarify during (t>t .^st few m.nutes how long the interview wiii 
last, what will be covered, and whsi .s expected Although th*5 gi^^und may have 
been covered in a pr einterwiew cWversauon as mentioneo ean<er. it a good idea 
to reviuw this information al me start of the actual mten/iew 

An important aspect of me interview ot orai exam, and one which is often 
ign<xed. is how 10 ask que* ^0^5 so that ?^e student has maximum opportunity to 
demonstrate knowJedge competence ^ut cuos from the interviewer The fol- 
lowing suggestions may prove useful m thi^ *^afd 

♦ Choose positive wording 

« Beginning wim me first quost^or es!a!>.&h ir^e pattern o^ the student s 

doing most of the talking 
« Make use of the calculated pause t yOu oon i say anything, the student 

will often elaborate 

• When possible, use a comment instead 0* a question Try to Keep the 
interview from looking hke a cross-examination 
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• Phrase quesUons carefully so that they are clearly understandable, so that 
you doni suggest the answer you re looking for, and so that you d6n*t 

j cause the student toJose face. 

• Always have a clear purpose m mind when you ask a question. \ e . don't 
asi(.unnecessary questions. 

• Us.e examples to help eliat more complete responses. Many students 
need that additional assistance m developing a framework or context 
within whteh to place their answers. Such examples may be included in the 
original outline or added for the particular situation. 

. .AlthougtLa-time limit is usually set for the interview as a whole, it is not 
.necessary to spend a specific amount of time on each section Sucfi flex»b»' ty 
allows the inierviewfat freedom to direct the questions in such a way as to ehcit the 
greatest amount of information possible in those a^eaa of greatest competence 
and to choose those questions m each category that are especially appropriate to 
the particular student's own expenences. Consequently, some questions included 
jn the origina! schedule may not be used during the actual administration Adjust 
ments may also be made m specific questions to make them appropriate to the 
student's level of sophistication in each category of knowledge 

Questions can be used to remind the student of omitted parts of earlier re- 
sponses, to get further information or probe more deeply, or to clarify the meaning 
of earfw remarks. U iS^ the responsiljility of the interviewer to help the student to be 
as definitive and speafic as possible and to s-.anfy ambiguous responses Check 
ing on answers that are unclear to the interviewer is important, t>ut it should be 
(tone without the use of leading questions that reveal the desired reply 

Another important eiement *n the interview situation is control. T^e interviewer 
needs to maintain control of the interview m order to utilize time efficiently and 
economically and to insure proper baiance and adequate coverage z' each a^'ea 

Control can be maintained by. 

^ • Being systematic— for exampie. by following an interview schedule care 
fully. 

• Exhausting one area before going on to the next (if you find you have 
forgotten something, do not interrupt and go back *n the middle or another 
area). 

• Paang the Interview, allowing enough time, not dawdling. 

• Avoid awkward pauses, although calculated pauses can be used judh 
ciously. 

o Focusing attention on the issues m question 

• Pushing the stuaent along with a question if he or she goes into too much 
irrelevant detail or gets off the track Antiapate the point where you can 
mtemipt, and do it with what Fear* calls lubrication, i.e , positive com- 
ments. 

The interviewer can further neip the student by being responsive, for example, 
by really listening, by shoeing that he oi she *s really listening (b> nodding. Sciying 
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^p^:^^,^[\Bpe"^\c,); by making use of facial expressions, ipton^gn*, ^^stures. postures, 
tli^^y 9'^^ frequent "pats on the back" or supportive coTijmente^hough it may 
necessary to tell the student (hat an answer is w;png, one shculB^avpid iettmg 
(/^v - '"^©^^w degenerate «nto an argument, ■^is.sufjffxjrtiveness generally results 
V\'L^^f^ ^ rnore qomprehenslve response Jrpm the student and consequently a more 
^^^jjjthorqugh assessment. -a ^ ).,v^> ^ 
^1 t\13j®''® ^0 schools of thought as to, whether or not notes should be taken 
dunng;^ interview. The argument against note-taking is that It Increases student 
, ^ anxiety afmj.can leadj^to loss of rapport, jn the case of an assessment interview, 
% 7^ howev/er, ifj&^rfis:^bsoIuteIy essential that notes be taken for the sake of accu-/ 
V/ racy. The risk of infereasing'anxiety or losijig rapport may be lessened by telling the 
^ ^'j^^ student in^fJySnice that notes will be taken. Tape-recording the interview is soi^ie- 
^* tmes :s|jtetij[ute'dior or used irr addition lio note-taking and has the advantage of 
^ enabllng^lj^a^tipn^ assessor to evaluate the outcomes of the interview ^er rt is 
^ ^comfiietw^:4tj§^^ to notify ^the student before beginning the inte/vlew if a 

\ tape recoc^itiS^Vjil tfe rniae. 

1 Pitfalls to^i:v&Cd' / 

Because Interviews and oJher oral assessment procedures are, mevitaoly affected 
by the personalities of the people involved, they are particularly vulnerable to the 
problems associated with validity add reliability. Although there is no way to control 
concpletely the subtle effects of iQterpersonal interaction, the interviewer can make 
more objective decisions by keeping in mind the sources of en'or in assessment 
discussed in Chapter II as w^l as the following pitfalls. 

The greatest danger in bral assessment is that some sort of interviewer bias 
willcreep In and affect the outcome. In addition to the halo effect discussed earlier, 
stereotypes and Interviewer expectations are another source of l/ias. The inter- 
viewer must be uarefu! to avoid jumping to conclusions and prejudging a student 
(either favorably or unfavorably) on the basis of any characteristic. Here are some 
examples: 

• Age. The interviewer may be reluctant to i^ive credit to a student who is 
only 1 8 or predisposed to cive credit to one who is 40. It'is easy, but wrong, 
to assume that since the latter is more than 20 years older, he or she 
automatically knows more about a given subject or has reached a higher 
level of competenco at whatever is being assessed. 

• Sex. Traditional male and female roles should not stand in the way of fair 
assessment, for example, a man may legitimately seek credit for expen- 
ence as a nurse. 

• Race. 

• Past experience. If a middle-aged student claims 10 years of expenence 
as a volunteer social vorker, his or her competence stii' needs to be 
assessed before credit is awarded. In other words, credit is to be awarded 
for knowledge, not for expenence per se. 

• Dialect. 

• Voice quality. 
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• Pronunciatior* (or mispronundaWon), ^ 

• Posture. ^ 

• Stance (for example, aggressive or meek), 

• ^pearance. If dressing^ neatly is not related to the knowledge being as- 
sessed, then a sloppily-dressed student should not be penalized for his or 
hor appearance. 

In addition, the interviewer should beware of prejudice about the area in which 
the student is requesting credit The male nurse example cited above could fit this 
category as well. Finally, the interviewer should be sure that \heimpficit long-range 
use of the assessment does not affect the interview^outcome. Assumptions about 
the use to which the student will ultimately put the assessment may be false. If, in 
the ODntext of the individual institution, different standards are appropriate depend- 
ing. on the long-range use of the assessment, then the assumptions should be 
,diecked and the long-range goals clarified. It may in some contexts be appro- ^ 
priate, for example, to have different artistic standards for the student who aspires 
to be a professional painter and for the student who plans;to work in advertising. 

In addition to interviewer bias, another pitfall to avoid in oral assessment is 
misurtderstandlng. If the inten/iewer feels at any time that there may be a misun- 
derstanding, on either side, he or she should check and clarify. Misunderstanding 
on the part of the student can be minimized by careful and clear wording of 
questions. A final caution'— the inten/iewer must avoid letting the student take over 
the interview. 

ANALYZING THE OUTCOME OF THE INTERVIEW 

After conducting the inten/iew, the evaluator or panel should review the student's 
responses to individual questions or to sect'ons of the schedule In general, it is 
helpful to arrive at a summary decision regarding the level of knowledge de- 
monstrated and the overall quality of the student's discussions. 

The level of student competence will, in most cases, vary from item to item 
and from area to area. One of the most difficult tasks of the assessor is to balance 
these vanatrons and arnve at a summary judgment. It is helpful to support overall 
evaluations with specific, representative examples of stronger and weaker re- 
sponses. These specific examples will also be useful in discussing the results of 
the inten/iew with the student. 

Some examples from an actual assessment inten/iew may help to illustrate 
this point. The examples were taHen from a structured interview developed to 
assess teadership skills. The inten/iew schedule and rating scales used were 
developed around 12 dimensions of leadership and are appended to the end of this 
chapter. How the assessor can back.up a summary evalualion is shown in the 
three examples which follow. 

1 . The student was able to see the relationship among the components of this 
competence area. Consequently, responses combined information regard- 
ing several areas. The inten/iewers had the flexibility, then, to eliminate 
questions that had been answered in another context and to encourage the 
student to draw comparisons among responses when that was appro- 
priate. 
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2. ft was af)parent that her levels of knowledge and skill varied among the 12 
dimensions. She was clearly at the evaluative level in the use of leadership 
skjils but had only basic knowledge of external power structures. She will 
need to complete additional study in this area in the future. 
. 3. One of the most outstanding q'lalitles of this student's responses was her 
ability to analyze, in ways new to her, her own past behaviors anu evaluate 
_their outcomes. She was then able to make recommendations regarding 
prodedures that might have been used differently or operating styles of her 
own that she now feels might have been made more succgssfuL 

'Following;a review of the student's responses, the rating scale should be 
completed, .and comments, either for the student or the advisor should be pre- 
pared. Whenever possible a copy of the completed rating scale should be given to 
Jhe student. Next, a judgment must he made regarding the amount of credit or 
fecognltk>n the student will receive, although some items in a schedule will allow 
for various interpretations by judges. However, tryout activities can help predict 
where variations may arise and permit some preparation of evaluators. 

Institutions may adopt different approaches or criteria for awarding credit or 
forrnal recognition of competence. A comprehensive discussion of guidelines for 
use in awarding credit is included in Assessing Prior Learning-A CAEL Handbook 
by Joan Knapp. Each of these guidelines js applicable to the analysis of the results 
of an evaluation interview. 

Following the completion of the interview process, the judge or evaluation 
panel should discuss the evaluations with the student. During this time the specific 
outcome— the level of formal recognition or credit awarded— should be made 
clear. Then, too, It Is beneficial to the student if some of the responsesJhem§elVes 
are discussed and areas for improvement or further study are identified. It is 
important that this discussion center on the criterfbn standards previously de- 
veloped and that it be clear that those standards were the bases upon which the 
evaluation was made. 



PROTOTYPE INTERVIEW SCHEDULE 

The following structured interview schedule and rating scale wore developed at 
Metropolitan State University. They were designed with the help of specialized 
consultants and reflect the context of this particular institution and the orientation of 
these consultants.^ 

Educational institutions that recognize knowledge gamed through experiential 
learning are often asked to assess competence in leadership of volunteer organi- 
zations. Consequently, this area was selected for the development of a model 
However, similar procedures could be used in designing interview schedules to 
measure a diversity of expenential learning, including more traditional academic 
subjects. 



5Motr6pol«tan State University expresses special thante to Sandra Halo, Bmer John, and Richard le»der 
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Prototype Interview for Assessing Competence in 
Leadership of a Volunteer Organization 



Goal Oifttdrmination 

1. What long-range and short-term goals form the basis of the princip^bi^niza- 
tion which you have served in a leadership capacity? ' 

To what extent were you a determining factor in establishing or redirecting 
those goals? $ 

2 In light of these long-range and short-term goals, what action priorities have 
been established? 

, Wha* has been your role in establishing these priorities? 
What factors do you see as the primary determinants of appropnate pnonties 
for directing the work of a voiunteer organization? 

3 Is there ever a reason for disbanding or substantially altering the objective^ and 
functions of an organization such as this? 

If so» under what circumstances? 

To what extent have you been involved in such a procedure, and how would 
• you evaluate the success of the changes? 

Organizational Processes 

1 What do you see as the major problems to be considered in organizing a new or 
existing group operating on a volunteer basis?" 

2 What steps would be particularly essential m developing such a new organiza- 
' tion? 

3 If you havp undertaken the organization of a new groufJ. describe the activities 
in which you engaged and the decisions which you made during that process. 
For example, in what way did you determine and establish the central identity, 
purpose, and functions of your group? ^ 

4 In what ways do organizational procedures and general maintenance of the 
organizational identity differ between a firmly established organization and one 
which is in its foundation stages? 

5 In what ways do the various objectives and functions of a group cause divergent 
concerns on the part of the membership of the organization to arise? 

In what ways have you dealt with thes<» types of concerns in interpreting, or 
adjusting, the identity or central focus of the group's activities? 

Organizational Structure 

1 Describe the basic structure of your organization, including the various levels of 
leadership, their responsibilities, and any subcommittees, along with their func 
tions and the rationale for the formation of those committees. 

2 Explain the relationship between thiS structure ^nd the organization s k)ng- 
range goals. 

To what extent would or have changing pnonties caused adjustments in the 
organization's structure? 

3 In what ways does the size of the organization influence the development and 
maintenance of a specific organizational structure— for example, in what ways 
might responsibilities be divided between a governing board and the functioning 
staff of an organization'^ 
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4. How does this structure affect the actual oj^rations of the organization? Cite an 
example fiWyour experience ic suppdrt your point. 

IV. Constitutional Provisions 

1. Is your organization based upon a constitution or charter? 

If not, what might have been llie effect of such a document on the identity or 
activitids of your organization? 

2. How was the constitution or charter of your organization, if any, first designed? 
Have there been substantial changes since the formation of the organization, 
and to what do yqu attribute these changes? 

If applicabte, evaluate your, role in the writing of the constitution or charter 
,a"ncVor the by-laws by which the organization is goye/ned. 

3. What issues seem to you to be most significant in facilitating an organization s 
following its constitution and by-laws? 

Describe any situations, from your own expenence, which ne^obsitated in- 
terpretation of these written guidelines or problems ansing 'rom their use by the 
memt}ership of the organization. 

V. Leadership Patterns 

1. Describe what you consider to be your loadarshiD style in directing the organi- 
zation under discussion. 

What do you see as your most valuable leadership skill in maintaining the 
motivational level of the membership? 

2. To what extent does your mode of leadership use general interpersonal skills? 
Cite some specific incidents, if any, in which your use of such skiffs was particu- 
larly essential and evaluate the results. 

3. In what ways do leadership functions and styles need to be adjusted according 
to varying sizes and functions of organizations? 

Cite several specific incidents dunng your period of leadership of one organiza- 
tion which particularly tested your leadership model and/or skills. 
' How did you resolve these difficulties, and what were the outcomes of any 
actions taken? 

4. Assume that one memt}er of the group attempted to dominate all the meetings 
by directing the discussion and monopolizmg the conversation to the point that 
members were beginning to cease attending meetings. In what ways would you 
deal v/ith this, situation, keeping in mmd the objectives of the organization anc< 
the fact that it is a volunteer organization? 

What tools of discussion might be used jr^ such a situation? 

VI. Fiscal Management 

1. In the leadership structure of your organization, who has been gjven Ihe re- 
sponsibility for the financial management of the organization? 

What basic procedure^ were used? 

2. What are the chief sources of revenue for this organization? 

Explain the ways in which that revenue is denved and your role in that process 

3. If this organization were to face a financial difficulty, how would you go about 
gaining additional sources of money? 

What condition^ within the community would have to be considered? 
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How might various sources of revenue affect functions of the organization or the 
cer«al identity of its membership? ^ 

4. Are there any legal restrictions of which you are aware which would restrict the 
sources of revenue open to you or the fundraising activities in which you might 
engage? ' . 

5. Has your organization engaged In any types of fund-raismg activities? 
If so, describe briefly the chirf techniques used. What do you see tc< i:-^ the most 
crudal elements in complpilng such activities successfully? 

Why did you elect to use the activities which you have selected? 

VIL Communications 

, 1 What other types of publications were prepared and used by your organization? 
What was your role In their preparation? 
How would you evaluate their effectiveness? , , 
2. Which types of writing did^ you use while holding the leadership role m the 
organization? 

How might the types of writing or styles used vary with the size, type, or purpose 
of the organization? (For example, might the target groups for written fund- 
raising material vary from group to group?) 

3 In what types of public-speaking situations were you involved while performing 
your leadership functions? 

To what extent were other members of the giOup involved in similar types of 
speaking situations and why? 

4 What other forms of oral communication have you used while fulfilling your 
leadership role? 

What role did they play in your leadership style? 
How effective do you fee! your use of oral communication skills was? 
Vtli. Use of the Media 

1 To what extent is it feasible for a basically volunteer organizatron to use one or 
more of the public communications media in enhancing Its vyork? 

2 In what ways did you l .arious public communications media in connection 
with'achieving the goals of your organization? 

Evaluate the outcomes of your own uses of the media. 

IX. Uses of Resources 

1 What types of 'nformaiion was it necessary for you to obtain in your leadership 
capacity? 

How did you go about obtaining that information? 
Which sources were most helpful? 

2 Assume that you have been asked to seek an external source of funding, such 
^ as a grant from a pnvate organization, for the purpose of extending some 

special service to the community. How would you go about locating and evaluat- 
ing various sources of funding for your project? 

X. External Power Structures 

1 With what sources of decision makmg influence external to the organization 
have you dealf^ 
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Describe the levels of responsibility of some groups with which you have had to 
deal or which you have attempted to influence in gaming decisions favorable to 
your organization. ^ 

Within that structure, where did pressure groups operate, and in what w?ys 
have you dealt with those pressure' groups? 
What were the outcomes of your erideavors? 

2. In what ways might the types of influence which could be brought to bear upon 
deaslon*making organizations, such as the state or federal governnrer! or 
another community organization, change ^with the size or overall goals of any 
given organization? \^ 

3. What typos of activities have you used to maii^tain continuing relationships with 
such sources of influence? \ 

How does this type of maintenance differ from ^more short-term but incisive 
activities undertaken to influence these power bases? 

XI. Operating Arena ^ \ 

1 . Assume that you have decided to open a new branch ot^your organization or to 
expand its membership into a previously untapped area.A^hat steps woula ,ou 
follow in determining what target groups might be most .appropriate for this 
activity and what needs your organization might meet in potential members? 

2. Descnbe the specific civic arena in which your own organizaUon has operated 
What types of issues, concerns, and needs within the community^are you aware 
of which have influenced the dev^3Iopment and direction of your organization? 

3. Descnbe several community arer.as in which volunteer organizations might 
operate and explain the leadership rule which you would probably play in Itg^t of 
the objectives of that community group, the limitations of action available, etc 

XII. Staff Supervision* 

1. To what extent have you had secretarial or clerical help within your organiza 
tion? 

Has this been paid r». volunteer support*^ ^ 
What has been yoir supervisory style in maintaining a continuing relationship 
with clerical staff or other paid workers within the organization? 

2. What prot>lems are u iique to supervising volunteer workers in an organization'^ 
What approaches h'^ve you used tc ^naintain continuino interest and motivation 
in completing necessary ♦asks'^ 

3 What differences do you see m leading an organization which is supported 
1 1 'o'dntially by voluntop' work ana one in whicn much of tbo /-to-day work <s 
done by a paid staff? 
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Rating Scale for Use in Assessing Competence 
in Leaderphip of a Volunteer Organization 



Student Competence Area 



Assessor Date 



Instiuctions: 

1. Read the trtle and ail three descriptions before completing the assess- 
ments. 

2. For each of the items listed below, place a mark on the line at that point 
which best represents your assessment of the student's demonstration 
of that knowledge or'Skill. The vertical lines reptresent approxirpate divid- 
ing lines between 3 levels of performance on each Item. 

3. if an Item Is not definitely applicable to this student, you 'jmay omit it. 
However, please specify below that item the reason for Its inapplicability. 

4. Indicate specific strengths and weaknesses in regard to each item in the 
sections reserved for comments. 



L Goal Det«rmlnatlon 









Understands the 


Understands the 


Understands and 


organization's chief 


organization's goals 


applies goal«setting 


goals 


and can help set 


process; 




pnonties 


understands the 






organization's goals 






and can set priorities 



Comments: 



il. Organizational Processes 









Understands how 


Understands 


Understands and 


this organization 


general 


applies processes of 


was formed 


organizational 


organizing new 




processes 


organizations and 






monitonng 






established ones 



Comments: 
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111. Organlzitlonal Structure 









UndAr$tanc(s some 
sUucturaJ features of 
the organization 

Comments: 
IV. Constitutional Provlsic 


Has comprehensive 
understanding of the 
structure of the 
organization 

ms 


Understands and 
applies the structure 
of the organization 
and Its relationships 


Understands some 
characteristics of a 
constitution 

Co/77/77enrs: 
V. Leadership Patterns 


Understands some 
provisions of this 
organfealion's 
constitution, if any 


Understands the 
use of a constitution 
and Its effect on the 
directions taken by 
an organization 


Understands most 
facets of own 
leadership style 

Comments: 
Vt. Fiscal Managernent 


Understands 
elements of 
developing a 
leadership style 


Understands, 
applies, and 
evaluates own 
leadorship style 


Understands some 
of organization's 
sc jrces of revenue 


Understands 
organization's 
pnmary sources ot 
revenue and some 
management 
processes 


Understands 
organization's 
sources of revenue 
and fiscal 
management 
processes; applies 
this knowledge in 
determining and 
evaluating alternate 
sources of revenue 
and management 
processes 



Commenfs 
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Vll« Cemmunlcatlons 









Understands 


Understands 


Understands, 


general role of 


general role of 


applies, and 


communications in 


communications ' 


evaluates various 


leadership 


and identifies sorpe 


types of 




of own skills 


communicatior^ in 






leadership 



Commonts: 



Vm. Um of MMlla 









Understands some 


Understands and 


Understands. 


uses of the media 


applies some 


applies, and 




techniques of us^ng 


evaluates the values 




the media 


and techniques of 


9 




using media in 






achieving the 






organization's goals 
o 



Comments: 



tX. Use of Resources 



Understands some 
major sources of 
information within 
the community 



Understands the 
value and 

techniques of using 
several major 
sources of 
information 



Understands a 
vanety of sources of 
information; applies 
this Knowledge m 
locating and 
evaluating 
community 
resources to 
complete a task 



Comments' 
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X» ExUauS Powtr Structuras 



ynderstands the 
oxlsteoco o< external 
power structures 


Understarxls 
orgai^zation of 
somo exfema) 
power eources 


Understands 
suucture arKi rote of 
external sources 
and appfies this 
urK5et*siancfir)g in 
relation to own 


Comments: 






XL Opereiting Arena 






Understancis 
general arena of 
own organtzatk>n 


Understands 
seve'a* possitTfo 
arenas for sucn 
ofgortuations 


Understands the 

csv^ arerus In wtik:n 

voiunteor 

orgarUzatkyts 

operate: ap^es tnts 

Knowledge Ijy 
1 analyzing arervi tn 

wtikrfiown 
1 organtzauon 
1 operates 



Commonts 



XM Staff Supervision 









Understands some 


Undersiaf)ds some 


Understands and 


tecfiniques of 


differences m 


appft .oChntques 


supofviston 


supervtsir>g 


^ supervtsing 




volunieef and paKJ 


volunteer and 




staff 


support staff 



Comments 
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S V a Product Assessment 



Ruth Churchill 

In many cases the most direct way tu evaluate the lectrntng that js either j cit or 
explicrt in a given experience is by means of an appropriate product or products 
Thts IS dearty i.^e situation in the creative arts Students Uaimmg sMIs m the visua! 
arts, for example, should have produced paintings, sculpture, photographs, films, 
or other art forms that g»ve evidence of their skills Writ.ng and rnus<cai composition 
are other examples ot arts in ^htcn the product ttseif »s important as evidence of 
learning. In other areas thejnr.portance of the product rray be less clear cut, but id 
many fields products are evidenses* of the learntng of skills and understandings In 
scjence, exhibits and demonstrations are reguiany used .n soence fa^rs as bases 
for making judgments about the leam.ng of such objectives as scientific methods of 
thinking, the ability tc present soentJic data cie"iy and to apply scientific ideas to 
new situations, and the tecfin«ca» skius tn constructing equipment effectively and 
safely. In teacfiing. a curricuium uOtt de^^eloped to teach certain Knowledge, under 
standings, or skills by a particular method to a given group of learners is a product 
that reflects an understanding of educ-atjonai phiisophy. a sensitivity to the needs 
of loca* situations and individual students, an cwarene:»s of current curncular ap 
preaches, and the ability to apply these to a particular problem Reports are impor 
tant products in a *arieiv fields atx)ratory and research reports and case 
studies jn the fieid of soctai work d't examples The iist could and should be further 
extended for example the evidence ^ri certain managenient situations comprises 
memoranda letters and reports that constituied part of the way »n whtch a situa 
tion was handled 

Product assessnr\enl ca^ u^jecJ wit^^ both sponsored and nonspo'nsored 
learning experiences Wa^ spo' '^^ored iedrnrng experiences th.:> method has the 
advantage of btmg planned iri adv^irtce to demonstrate relevant skills knowledge, 
and understanding^ in a tas^^.c^^ i^ai cari t>e evaluated fa.rl> Altr^ough certain 
elements of the product car be ijo.qoe * - eac^ student ii iS possiDle to stress the 
communaifties givir>f; a" opporton(*y to pian 'efaDie and economical means o^ 
evaluation Products --r^sponv^red prio*- lecirning experience du not have the 
advantages of piannir^g tjy \r e^ ♦^eqiie y ^n^e advantage of Pemg one ot the 
few sources o* exter'^a' e. oe^ce a^a lao!*^ 

Some products unsponsorec; teaf^-nq expe^^^ences Sitf. as worn ,n the 
Cfeativt aris may occo' w^t^ suc^ sut^'t'en^ 'ego.anty triat a working procedure for 
pfoduct assessme'^t wctrthw^j.ie T^'t* w j^k um^ a few '-luOt^nts is eva^ujted at 
any one t»me Du* o^h^ a pe^'^oi o* rmf a "etat've^ uvqf^ riumber of sluoents 
products wtii i^ave t)e evaluate ' i'' '^^ef ..ast'S the producS subntitied will De 
diverse and *orma Tted^s of as^s^^ < pf-oducts wousc lake more time and effort 
than would be iustjf*e:3 Nevef'^e^e^ " ways whlc^ these d. verse and torrc 
times UHtQ p'o*3^c's at** r-^^V'j ) c--!'" or* '^v.jnr.ced i^p oo^ts<derations 
procedu'es a'^-cj . ' * ' ir*»'' 



O ( 



Product Assessment vs. Performance Assessment 



Perfomiance testing and product assessment need to be distinguished. In perfor- 
mance testing the student actually pe/fom:s a gwen skill, such as playing a musica! 
composition, in product assessment, all that is available is the en6product, m this 
case, perhaps, a tape or a record of the student s performance. Whether to apply 
performance or product assessment depends upon a number of factors. 

First, which is more important, the performance or the product? In the example 
given, the emphasis is on the performance. In the case of a musical composition, 
the product rather than the composer's particular method of composition embodies 
the important skills and understandings. However, even when the emphasis is on 
performance, a product may embody it in a useful way A tape of a student playing 
a given composition can be regarded as a product, for example. This would allow a 
student to select the performance he considers hts best, and might be appropriate 
as an Indicator of piior Iea»'ning On the other hcind, a tape made under structured 
test conditions with observers present might more properly be regarded as a 
performance test, and would be more appropriate if the ability to perfo.m before a 
critical audience is part of the desired learning. 

Second, practical considerations may determine the use of product assess- 
ment rather than performance testing In some cases, the uses of products rather 
than performance may be more convenient, less expensive, and more reliable. For 
example, if several students make recordings of a musical composition (under the 
same conditions of recording), a jury can make compansons and replay parts 
about which questions or differences of opinion arise And, of course, in certain 
cases, especially m nonsponsored learning experiences, only the product is avail- 
able. 

Product Assessment and Written Material 

Another distinction (or case of overlap) that may have to be made is between a 
written product and a written report or paper or examination. Obviously, if the 
purpose ij \o evaluate ability to write effectively, the methods foi assessing fcee- 
response written material would apply A paper \A/ntten to demonstrate knowledge 
or understanding of a field would probably be evaluated as such papers usually are 
in academic situations On the oiher hand, imaginative writing, such as plays, 
novels, poetry, can be judged as a product But there are other cases in which the 
wntmg was produced for other reasons, naturally m the cours« of the experience, 
and IS to be judged for the learning of other skills and ui.derstanding. Diaries and 
logs of experiences while traveling, case reports written by social workers, and 
laboratory and research reports are dll examples of written products which can be 
assessed as evidt^nce that specified skills and understandings have been learned 
In many cases. ,ouch products could be used m conjunction with an interview. For 
example, a travel diary might implicitly reveal growth m understanding of another 
culture, but such growth would need to be checked explictiy in an interview 

An example of the use of a structured daily log is given »n GAEL Institutional 
Report No 3 ^ Students at UCLA m a number of courses n which expenences 
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outside the classroom were incorporated kept daily logs in their field work or 
internships, In these logs they recorded both activities and reflections on what they 
thought they had learned. The logs were then analyzed by faculty experts for 
evidence of student learning, using a Process/ Assessment Matnx Five broad 
areas of ioarning— self-awareness, awareness of others, skill development, 
academic content, and career understanding— and five levels of involvement- 
identify, inform, descnbe, generalize, and apply decision-making skills—form the 
matrix. Thus, using the log as the product, the faculty experts could analyze it for 
evidence of student learning and of the lovel of that learning. 

When product assessment is based on written material in any of the above 
senses, the problem frequently arises of distinguishing between the product and 
the* writing. Much stress is placed in the academic world on well-written papers, 
despite the fact that student papers are frequently artificial They are written by 
individuals knowing relatively little about a subject for other Individuals who know 
much more. It is possible to place too much stress on this type of writing, poor 
products can be disguised by well-written reports and good products can be lost in 
poorly written papers. 

A useful way of meeting this problem is to ask the question. What kind and 
level of writing is demanded if a particular product is to represent student learning 
of knowledge, understandings, and skills at a college level? At the one extreme, a 
novel or a series of poems must be judged on the quality of their writing At the 
other extreme, the niceties of writing are not vital in a daily log. In between there 
will be a variety of possibilities In a study of curriculum units judged as products, 
one of the best units presented contained no formal summary of ^hat the student 
had learned from teaching the unit, instead each page of the unit was annotated 
with comments on what had and had not worked and how the unit should be 
changed before it was used again The writing involved m products should suit the 
necessities of the situation m wh.ch the product is to be used rather than notions of 
academic propriety. 

Product Assessme^ and Documentation 

An important and necessary distinction needs to be made between product as 
sessment and documentation Students frequently present documents, such as 
letters from employers, newspaper clippings, licenses, and other materials, as 
evidence of experiences Rarely do these afford evidence of learning, or if they do. 
of learning related to specific, relevant objectives It is possible that a selection of 
documents produced on the job by a student could be the product equivalent of an 
in-basket lust, but the skills, knowledge, and understandings to wh.ch they were 
related would have to be specified and evaluated In most cases these products 
would need to be supplemented by an interview 

Product and Process 

In many cases products cannot stand alone without some idea of l..e processes 
v/hich produced them When the ^loduct aoes i>tand alone, the bur Jen is on the 
evaluator to identify the learnings represented In the model of product assessment 
for the visual arts, three criteria have been identified, representing generalized 
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learnings in this area; but even in this case, a short paper or interview has been 
suggested to allow the student to clarify his inter^t in some of the work he pre- 
sented. In the UCLA analysis of logs the matrix of criteria for learning has already 
been worked out arxJ includes a process component. 

While the product can testify clearly to some of the leamings, leamings as- 
sociated with the process by which the product came into being need to be 
evaluated in other ways. For example, in evaluating ability to develop curricular 
material, many learning? are implicit in the unit itself— for example, ability to set 
objecHves fpr- students, to select materials, to evaluate changes in student be- 
havior. Equally impgctknt process leamings are not explicit— for example, how the 
Student identified tKe problem Initially, how he gained the background needed to 
work on it, how he evaluates the success of what he attempted to do. Here tKDth the 
product and the student's account of how he produced it are needed. 

Developing Product Assessments 

A brief sketch of the steps to be taken in developing a procedure for assessing 
products may be helpful for faculty members faced with the task of planning formal 
or informal procedures for evaluating teaming from the evidence contained in 
products The procedures suggested represent attempts to increase the validity 
and the reliability of the judgments to be made. 

1. Identify Learning Outcomes. The first step to identify the learnings ex- 
pected of the student in the situation Once these have been identified, perhaps 
with the student's help, tt should be clear whether an assessment of a p^-oduct will 
yield useful evidence of the quality of important leamings or will need to be 
supplemented by other methods of evaluation or will be inappropriate, A novel may 
stand by itself as evidence of the learning of skills in imaginative writing, the 
accounting records of a business venture may need to be supplemented by an 
explanatory paper or interview to «jentify and document Jurther the relevant leam- 
ings, but what product will represent skJIs in interviewing— unless tne student has 
taped several interviewing sessions'? 

2. Construct a Form for Product Assessment. The learnings already identified 
will constitute the criteria for the product assessment Not only should the critena 
be specified but also the relevant points to be observed m the product. If at ail 
possible, levels of quality should be indicated in terms of characteristics of the 
product that are indicati/e of different levels of performance, For many products 
the best approach to defining critena and products will be to provide specific 
examples of outstanding, averaga. and poor products 

A simple method is to construct a checklist, using each identified learning as 
one item on the iist For each item, a rating scale can be devised to judge the 
quality of the student s oerforniance The more clearly the rating scale is anchored 
in observable behaviors, the more consistently different judges can use it At the 
time of construction, decisions can also be made whether certain learnings are to 
be given more weight than others m arriving at a summary evaluation. 

The two models at the end of the chapter illustrate different ways in which 
formal assessment procedures can be set up, the PrQcess, Assessment Matrix m 
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GAEL Institutiop^ Report Nc. 3^ gives stiil another possible variatior^. For infor- 
mally assessing a unique or infrequent type of product, a simple list of leamings to 
be checked for the adequacy with which iriey have been achieved will help insure a 
consistent and a documented approach to product assessment. 

3. Prepare Students. Guidelines need to be written for the student so that he or 
she understands the criteria by which his oi her work will be judged. The student 
also needs instructions for prepanng his product for evaluation, highlighting the 
important features of the product, and eliminating extraneous details which might 
adversely influence the evaluation. This step will usually not be difficult for spon- 
sored teaming experiences in which the product can be planned. Although more 
difficult for prior teaming, such instructions are worthwhile even if they have to be 
modified in special cases. 

4- Prepare Evaluators. Judges or evaluators also need guidelines for observing 
and judging products. In most respects these do not differ from the guidelines 
given students. The most important guideline for the evaluator is the form for 
product assessment. 

If more than one evaluator is to be used, either to obtain more than one 
judgment of a product or to have several faculty availabte to work with students, 
training sessions using the evaluation method on a few sample products are useful 
to identify disagreements and permit discussions leading to rtJvisions of the proce- 
dure and agreement on decisions. 

5. Combining Product Assessment with an Interview or a Paper. Another part 
of planning the evaluation consists of deciding whether or not to combine the 
product assessment with an interview or a paper. Such a combination may be 
necessary if it is not immediately clear that the product gives evidence that the 
student has learned a certain skill In addition, a product may revaal only some of 
the learnings possible in a given situation There is the danger that ability to 
verbalize about the skills invoUed m a product may not be the^same as the ability to 
make the product, and m the interview or paper some students may be rewarded 
for verbal skills rather thai fo those for which they are presumably being 
evaluated 



A MODEL FOR PRODUCT ASSESSMENT IN CURRICULUM 
DEVELOPMENT 

To illustrate the possibilities of product assessment, two different plans for asses 
Sing products are presented The hrst model is adapted from one developed for the 
master s program in education of *tae Juarez Lincoln Center of Antioch College it 
was developed m a study of assessment procedures m the CAEL Operational 
Models Project This model consists of two documents, available to both students 
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and faculty. The first is a statement of cntenon standards for projects In curriculum 
development; the second, a checklist for evaluating such projects. 



Program Statetnent in the Area of Curriculum 

To meet minimum requirements the student must demonstrate a general Knowledge of 
educational philosophy and its relationship to various curncular approaches The stu 
dent must be able lo express this knowledge in terms of hi&her own educational 
philosophy and the goals andor objectives ^state guidelines, local goals, etc ) to which 
the Curriculum unit is related. 

The development or revision of the curriculum unit is expected to include general 
f^tionale, specific objectives, method of teaching and learning, data sources provided. 
^ and Tittar^s of evaluating changes in pupii behavior produced by the cumculum unit 
m general, the student iS expected to accompany the curnculum unit developed or 
revised with an account of how he or she identified the problem, acquired background 
for handling it, earned out the project, and evaluated his. her general learning from the 
project. 

For major credit, a student must also teach at least part of the curriculum unit, 
provide a staff guide, evaluate the success of the curriculum ^.iit in terms of pupil 
learning, and relate the project as a whok? to other areas of professional dovetopment 



Checklist for Evaluating Projects 
in Curriculum Development 

Name of Student . . Name of Project 

Educational philosophy and goals 

1 Does the curncular unit reflect the students educational philosophy'' 

„ Yes. clearly 

Yes. m some respects 

Questionable 
No 

Evidence {How does the student indicate hts philosophy'') 
Space left for free answer! 

2 Does the project relate knowledge u' ^ .Jucational philosophy to possi 
ble curncular approaches^ 

Yes. clearly 
. - _ - Yes, »n some respects 
Questionable 
No 

Evidence (Wnat curncular approaches were explored'? How is the 
curncular approach chosen related to dducational phii- 
sophy'') 

(Space left for free ar^swerl 
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3. To what goals, objectives, or guidelines has the curncuium una been 
related? 

State guidelines 

Local goals and objectives (school or organization) 

Teacher's goals and objectives 

Pupils' goats and objectives 



4. Has the curriculum unit been related to the goals, objectives, or 
guidelines chosen? 

Yes, clearly 

Yes, in some respects 

Questionable 

No 

Evidence. (Cite specific objectives and show relationship to curncuium 



Development or revision of curriculum unit 
5. Is the curriculum unit cleariy described and presented'^ 

« Yes, clearly 

Yes, in some respects 

Questionable 

No 

Evidence: (What is the subject matter o* the uni**?) 

(Space left for free answer] 



6. Is the rationale for tha curncuium unit cleariy stated'^ 

Yes. clearly 

Yes. in some respects 

Questionable 

No 

Evidence (What is the rationale'^) 

[Space left for free answer] 



7 Are the objectives for the curriculum omt clearly stated'' 

Yes, clearly 

Yes. in some respects 
Questionable 

_ No 

Evidence (What are the objectives^) 



B What IS the method by which the curriculum unit will achieve its objec- 



unit.) 



(Space left for free answer) 



1 Space left lor Iree answer] 



tives'^ 



[Space left lor free ar^swer] 
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9. Is the method appt^pnate for achieving the given objectives'? 

Yes, clearly 

Yes, in some respects 

Questionable 

No 

Evidence: (What methods are used, how are they related to the objec- 
tives?) 

[Space left for free answer] 



10. What sources of data are provided m the cumculum unit'? 

(Space left for free answer) 



11. Are these data sources used adequately'? 

Yes clearly 

Yes. in some respects 

Questionable 

Mo 

Evidence (Give an example of a data source and how it has been 
used in 3 unit ) 

[Space left for free answer] 



12 By what means will the success of pupils in achieving the ob]ectives of 
the curriculum unit be evaluated'? 

[Space left for free answer] 



13 Are the methods chosen suitable for evaluating this curncular unif? 

^ Yes. clearly 

Yes. .n some respects 

Questionable 

No 

Evidence (Give an example ^ a means of assessing what pi pils have 
learned 1 

{Space left for '^oe answer) 



Use of the curriculum unit 

14 Has the curricular unit been used*? 

Yes. the whole unit has been tried out 

Yes, parts of the unit have been tned out 

Questionable 

No 

Evidence (What has the stuJent done to put the curncular umt into 
prac^ce^ 

{Space left for free answer] 
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15. Has a teacher or staff guide for this unit been prepared? 

Yes, one which can be used as it stands 

Yes, one wh;ch could be used with revisions and'or more work 

Questionable 

No 

Evidence: (Could another teacher use the guide as \\ now stands'?) 

(bpace left for free answer) 



16. Has the use of the curncular u.nit been evaluated aopropnctely? 

, Yos, clearly 

Yes, In some respects 

Questionable 

No 

Evidence (Cite meanb of evaluation, appropriateness, and adequacy 
of conclusions drawn } 

(Space left for free answer) 



17. !s the area of curriculum development related to other areas such as 
development of sources ol information, staff development, evaluation, 
community invotvemont, and bilingual- multicultural education'? 

Yes, clearly 

Yes. m some respects 

Questionable 

No 

Evidence (Space left for free answer) 



General 

18 What sources of data have been used tn carrying out ihis projecf? 
(Space left for free answer) 



19 Have these data sources proved adequate tor the project'? 
Yes. clearly 
Yes. in some respects 
Questionable 

, No 

Evidence (Background tn educational philosophy, curriculum de- 
velopment, specific area of curriculum development j 
(Space left for free answer] 
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20, Is the report written at an acceptable level for an educated adutt? 

PubtishaWe c 

Adequate educated wnting 

Nee<^ work on wnting 

s Writing fails to communicate 

EWdence: [Organization, clarity of presentation, use of words, gram- 
. mar.) 

[Space left for free answer] 



21 . As a whole, are you satisfied witn this curriculum unit and the report on 
its development? 

Major Credit* Mino: Credit" 

Clearly satisfactory 

Adequate 

Questionable 

Not acceptable 

*Pay no attention as to whether project was intended to receive minor 
or major credit. Rate it as to whether you believe it should receive 
major or minor credit Major credt implies that all items have been 
carried out, for minor credit the curriculum need not have been used. 

Evidence. (Outstanding strengths and weaknesses of the report.) 
[Space left for free answer) 



A MODEL FOR PRODUCT ASSESSMENT IN THE VISUAL ARTS 

This plan Is adapted from the CEEB Advanced Placement Test .n Studio Art,^ 
probably the best current example of product assessment based on research on 
judgments by art faculty of student work m art. It can be used as it stands, modified, 
or altered according to the purposes of the assessors. It can also serve as a model 
for further exploration of ways of assessing products of students. 

Instructions to the Student 

In order to earn college credit in the area of the visual arts—pamting, graphics, 
sculpture, ceramics, photography, film, tele.*oion, and other forms— you are asked 
to present for evaluation selections from tr.t; work that you have ddne. The purpose 
of the evaluation is to allow you to demonstrate your ability to aeal with fundamen- 
tal concerns in the visual arts. 

The materials that you are asked to present deal with three objectives, overall 
quality of work, depth of work, and breadth of work. Evaluators are looking for 
works that reveal a style of execution and content initiated by you personally. The 
works presented may have been produced at any time on your own initiative or m 
art courses or other courses The only restriction .i> that they should not be work for 
which you have already received college-level credit. 



•Co<teg« Entrance Exam^naboo Boa'd Advanced ^-cccr^c^: Ccjrzc Ocsc'pi^^ Art 
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In the msimctions that follow, certain instructions are given on the works you 
are to present and the ways m whjch you present them The purpose of these 
instructior\s is to help you present a good sampte of your worK on the same basis 
as other students,. In that way you can help the judges make a fair appraisal of youf 
leaming. 

Overall Quality of Work. You are askeJ to present, tneir ongmai form, three to 
five works which you consider your best There works will be judged for theif 
overall quality. They may be individual works, or they may form a series All of the 
works may be m one medium, or they may be in different media. The reason that 
you have been asked to submit a few of your best works is that experience 
indicates that large r umbers of works include pieces of varying quality, and judges 
tend to evaluate the 'otal groi<p at a lower I'-.^el than thwy would if a careful 
selection of a few works had been made. 

• If the work, such as a filnu «as r.^ne m collatx)ration witn other individuals, 
specify what were your responsibilities and contributions ana what were those uf 
others. 

• Since these are onoinai works and since they may be handled by sevef<ai 
judges, try to keep the porJono manageable in size and to protect the works from 
the consequences of unavoidable handling Graphics that can be smeared should 
J.ave a protective, transparent covering, small pieces, such as jewelry, should be 
securely mounted, tapes and film should be adequately packaged 

• Remember that the better you present /our works, the less i»kely are the 
judges to* be Distracted by irrelevant considerations 

Depth of Work. To give you an opportunity to show yOur personal concentration 
on a particular way of thinKing, workfhg. and producing you are asked to present 
evidence related either to an aspect of fOur work or lu a project on which yOu ha*e 
spent considerable timo and interest and w/?iC/7 you may jr may not Qons*d3r 
successful Be sure that the material shows work on a particular act ♦?/ or investi- 
gation carriea out over a considerable penod of tirrte Remember that the wo^k 
should be one that yOu were personally committed tu execute Class problems »n 
drawing or design, for example, are generally not acceptable 

Although the work may stand on its own merits and require no further clar»fica 
lion, you should present a short Antten or taped commentary to -iccompany the 
wr/rk It helps the evaluators to Know why yOu selected this area or project, what 
influences (things seen, heard, read, feit .magined> affected yOur work, huw pian 
ntng affected the development jf your worK. and ^hy yOu feei tt is successful or not 
successful lAlternatively You will be asked in an interview to tell the evaluates 
why you selected this area or project Ahat mliuencea ith.r»gs seen, heard, read, 
felt, imagined) affected your work howpianmng aff'^cted the development of yOv> 
work, and why you feel it is successful or not , If your work is part of a group 
project, your responsibilities should be dearly defined 

Most works for this. section should not be Submitted in their onginai form ^.th 
the exception of illustrated books and films including videotapes Sculpture oamt 
tng, murals, graphics, ceramics jeweify weaving, photo^^raphy and architecture 
may be illustrated by coic sude^ Environmental or .onceptuai projects can be 
de'.rnbed by sndes. films and or appropriate written documents Take care m 
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photographfng your .work, Tho quafity of your slides tm ic be good sc that judges 
can evaluate your v^ork accuratefy. Fot IhreeKfimeosionai worx. provide severai 
slides'for^ach work, each sfide showing a different view of the work Try t& keep 
your presentation short {no more than twenty slides or five mmutes of f.^i 

^.BrMdth of Woik. Here you are given an opportunity to show the range of prDt> 
i^ynis, liteas,. media, and approaches you ha^^e undertaken You are asked tg 
submtt slides of completed works that show how yuu have so^ed proo^ems in the 
4^wfng Categories: 

• spatial Illusion on a flat surface 
' • color 

# drawing 

IP organization of three-dimensional figures 

Or) the slides identify the prcbiem you are attempting to soNe for example, 
tse of space or oo\ot relatjonships. This secton iS or>e wh»cf e valuators K)Ok at 
aspects of your work jn coio design, and tech.nque ^^waiiveiy .&o*dtea from She 
total quaffty of the work or from wtuil you wished to expiess ^e*preso*ve intent, 
Total quality always influences e^aiwators lodgments, but art skills are '-^ 
phasized in this secbon 

Sirx;e any work submitted wm probat>iy »ncorpoiate more U"ian one category, 
use works that clearly emphasize urie category ^yeft ihouyh uthei ^fobiems n^ay 
be present. Works appeanng »n this sectjj*/:. t.^^^ »j.»^HiCJ> have appeared tf. one J 
the first two sections. Make a ca/efui selector. shoe's Uy to Keep yOur presenia 
lion short (no more than twenty slides) 

instructions to the Evaluators 

Works of art are highly personal anu ^uu^eci.vb » c*-»f.tent ar»o expfess-^ medO:* 
They can only be evaluated if judges are firmry committed to I^ws ujntept The 
mos! important part of the assessment ^s the stu^dent 5 osponse O^e/a i Ouantf 
of Work. Quality is aiso an »mportdns part uJ the ju^lgment^ made ri [jepth of //orh 
and Breadth of Work 

Clearty the student must perform at cos»ege Jeve* if he of bhe s \o lecet^e 
credit Since professors maKe judgmenis as to vwhetr^e? o» rMji then stude»^ts are 
performing satisfactorily ir» tunege cuwf^es .n art A s assumed thai they van ,u<3ge 
college-level performance un the bas»s jf a cafetuiiy selected portfoiju J art »v«:Kks 
submitted by students whose work .n art iS based on either sponsuied ,\,u**en{ and 
planned) or nonsponsored ^pfio» and nonpiannedi exoenences 

The validity of the three cntend proposed iOvetan quai.i^ depth and breadt*^, 
IS based on research on the factors %^hich coiie^ .nslrucfo^ - 1^. art suhooJs *"a^f^ 
found important arxj at>out which the> coutd acjtee theit j^.>gments Vafedity 
the judgments is aiso affected Dy the seiectior J the judges ^t. '^jai?jf fudges w 
b© from the institution s own art faculty but 3o.t^ ..-31 *wrl.cns w H want to .ncudo 
practicing artists and art facutty from other institul»ons it is important that t^o 
judges review the enter ta and make su^e that thf ;}g.o^ w.1^ t^^r ,V ar: faculty 
may want to formulate »ts own cr.te^a both ^ terms now thoy state them and 
what works they asK stude'its to subm^' in irmt cas^ the p'ocedu- es i>et forth hete 
can be used as a model and adapted 



0 



At least two |udQ6S be uSed <n me product <ii^t>^?«^T of &itj«:^o art 
Uoi CKUy doos the u$o of mo^c snan ,udgo <ocfeaM? me ^ai^diiy tho iwdg 
meftls boc^uso ol bfoacJer t>auKg;o>jixJ .r art ,op<ei»enieo u*i .1 ti^3<. nue^ae ^ 

rty can also be troeased by «^ imi-ntn^ session Two i^w ^ampia J^Jersi a? 
stiAfe^^f wofk can ba tt\ac^ s*o. and e^fiiuaior w<ir i^dge !t>*>m ,«nliKHjndeni., 
Atlof iho jnctepOfKtenrjud^mofilb. ?^ve C^hja rr*<joK? ♦rKj»* do^/ «v^*et^rner»!s> is -*^ 
(^(jgreameols of iho jt/dge^^ tan bo ;j£i>c^^&od, me judg^^s ^af^ a?* as ar 
agfeomem 00 their tmi^ Th»tSk p^ovedufo ^.vcrs me jucjyos. ,^?Tie a4>5!^j.in^e tr^^ 

0^ swnto qua% Tola? agreement cf jt/do©^ \ >ot M^^>45^t Ow! ^aino^ a gone^a 

10 insure ihai au studot«ls. are jwdq<3d u^*>dit.v^ :> ai. *ai ab iA?>*»^«t*t* 

1 Judges sttcwJd w<kh 'ndopefv:Jentjy N^. viow an^ct^^0r^ 
scorns^ ursM l^^o jyd9i*s ha\t* ^fni^e<5 

2 The fcvork of |ii«>9e^» ^*>^>u*d t» :tCfiedusotJ ^ i rv^int*.* '^^OiJfd 
appro:t*malo«v ^c*^^^!^ ^^^^ M^r. _ -t e m^r..tti\*tr\ m<i 

J H a*'^ <^%fV'j(?v, »t u? ^r'-.l'i'-i >>;t''* v^-^r^ . »^<-^>,h ^ !ne 

flaiing Of Cv<j?B« Quality Of WoxV - : * v> ^^^'»* ' ^ i # . .i^*'-: 



stuc^nt submits up lo twenty sitCa-y »MjfK anr^jugf I'tr^b vK3tiOtdpei^ and »Hui> 
trated IXJOKs may also l>8 bubm^tied The p^ofe^t y'v^uics t>e dcxx>fnpanied t>y a 
short commenlar/ ta.- j why the &tj<}0nt ^e^ected iN* area p^^^ject wnat n^u 
enced the work, how piannmq tiffeclec tne de^efopn^fern oMr t? wo'^^ <ind whetNc 
ihe stiiCJent leets tho wor\ was ^jucces*;*^ ^ "o* AiMrrat.vt* y e ..Oge^ '*^a> 
wish to interview the studer^t o'* ihe^v jjv^^sts 

Important facto'^ tx? cop!Se'e<; - ,uc:q - ^ .n^ 

♦ Co<'»C€ntratHon <ipa" . 4' p'v. .<*• * ' t'* < ' r u? ; ^.♦•^ <a3 

time 

Rating of Breadth ot Work \ - a uOqr-^.' * Mivf i aspi»i:ts 

0^ the Sluderil ^ %V0'K r, ^^^f JO^,,^ ;J ^^^r SC'.<J?**'1 

total Quality ol Ihe wo^* S^.0^'^^^ ^ ;t • s led ^ s{ i* a . c-r-* 

Overall Rating, u s ^ ^ ^ -i^^. » ' ^ 

pfOduci as&es^irK'^t "\ r j • i . . • ^ • • ♦•vi* i' ^v*^ ' * 

<fer»t$ work Tf*/' ; J > *- ' • Jt', s * , ' . • \. ^^i^' 

the Ih*^^* ^itKt^orti . Vt , , • ' t-^ . * , ^ a ' At- t;*"' 

snc* fd De cop^PHiU' * : "v' " ^»'' t i ' " s' . : 'i^* ^v*- 

-i* ^*'«» ' i * . :* .V , / ; 1 « »' : ♦ : ,* 

n.nq .f'i.** < ^ * , ; , , /. " • ' " r 

^u<'k4'(' ' *^ " I * . : . ' . " * ' i s * • i' : i' I 

* , • A , ^ • . *. , 

^* i/f • A' , ' • ' • < • , * . , . 'A • 

< / " » ' <" . ' * . ^ ■ * 
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■ Performance Assessment 



Arnold Fletcher and John L D Clark 
NATURE AND BASIC ISSUES 

The performance test »s usually described as a test that evaluates a person s ability 
to carry out or execute an action, process, or task Although paper-and-pencil 
Items that involve the vvritmg of symlxjis in response to symbolic stimuli may be 
used, performance testing more characteristically stresses the ability to manipulate 
something or perform a role— for 3xample. typing, piloting a plane, assembling a 
piece of machmery. playing the violin, teaching, acting, or leading discussions 

Some Distinguishing Characteristics of Performance Testing 

In attempting to place performance testing m an assessment perspective, we must 
emphasize that much, but not a!!, of such testing involves the measurement of 
skills, described in dictionary terms as developed or acquired abilities, which are 
usually identified with psychomotor activities In addition, some educators (end to 
treat skill testing, performance testing, arid competenct! testing interchangeably 
Although related, they are nol riecessanly synonymous, using them synonymously 
can cause confusion about the concept of perforTian^e testing and can lead to the 
misapplication of testing techniques 

Kelley descnbes competence based education m terms of four categories of 
objectives (1) cogn/i/^e- what the student should know. (2/ a/Zecf/ we— beliefs, 
values, and attitudes ot the md.vidual. (3j per/ormjnce— specified behavior .to t)^ 
demonstrated by the learner and (4^ con^ec/tienue — student learning identified by 
the teacher' Thus a skill test tends to evaluate a more specific limited ability 
usually within a largei performance uunteAl. whereas a performance test geneially 
attempts to assess a higher level, more complex behavior, which in turn, is often 
part of an even more comprehensive competence 

Another distinguishing characteristic of the performance test is the presence 
of a large element of realism, or what is sometimes descnt^ed as situational 
realism The student is expected to perform in as close to a real-life situation as 
possible Since it is very often impractical or impossible to bet up such a situation, 
the assessor must simulate reality This ;.ractice is characteristic of student- 
teacher training programs, the type and degree of simulation being dependent 
upon the particular needs uf the student F.tzpatrick and Morrison indicate that for a 
simulated situation to be repieseniative uf a real life environment it must (a) be 
comprehensive- simulate a range of varying aspects of the situation, and (b; 
demonstrate ^c/e//fy— reflect a desirable degree of reality ^ 

As a '.nal note, it may be useful to reiterate the differences t etween perfor- 
mance and product assessmerjt Ab pointed out m the p'^evious chapter, perfor- 

6(J5a' Ke'-ef '^'e^ ^ • * ^ , r > »■ ^ . ^ ■* 



iRJC 



63 



mance assessment often resuits tn a product that can also be assessed. Where an 
end piipSuct ts a/ailabte. the emphaSiS tn perfornnance assessment is on the an 
tecederli steps performed to arnve at the product, rather than the product itself 
Systematic procedures for observing and recording behavior are usually charac 
tensticbf performance f.ssessment whether or not a product emerges. In addition, 
performance assessment procedures are usually characterized by externally struc 
tured conditions that remain constant for different individuals whose performance 
is being assessed. 

Types of Tests 

Classification jo any field poses the prob'em of overlap Tests vary so much in 
terms of purpose, content, and procedure that different authorities approach clas 
sification in different ways The categories, descriptions, and examples presented 
herein are not intended to be aii-molusive but, rather representative of the principal 
types in current use. The reader is also referred to the appropriate works in the 
bibliography for additional treatment, particularly the CAEL publicatiOr.-A Compen 
dium of Assessment Techniques by Knapp and Sharon ^ 

The work sample is one of the most common types of performdnce testing, 
usually possessing elements of simulatior. In this case, pb related tasks are set 
up. following which the student or trainee is ey/aluated for the task performed or the 
product created The work sample tends to recreate an important part, if not all, of 
the actual job tasks and operations, frequently usmg job equipment The test will 
sometimes involve certain on-the-pb difficulties, which the examinee must over 
come. Although som^ of the realism may be sacrificed, the critical job elements are 
present and tne lest is leadily recognized as exemplifying situational realism 
Referral to other GAEL publications, prepared for both faculty and student use, is 
suggested for further coverage of work sampie assessment problems ^ 

The situaUonai test endeavors to evaluate performance in a real life or simu 
lated situation m wnich the examinee is asked to undertake a task or role that has 
been thoroughly explained before being carried out and assessed Sucl^i t<.^ts 
usually attempt to evaluate more complex performance qualities, oftentimes in 
situations involving human interaction Some representatwe examples are the 
following 

1 War games and business games 

2 Pole-playing in supervision and maridgement crimindl justice and socuii 
work 

3 In-basket techniques m admimstfaiion and office practice ' 

4 Employer evaluation of employee performance 

5 Case studies in sociology and numan sen/icf- 

6 Simulated clinical problem solving m allied health ftefds ' 

1 A ' '■xJ*^ »' » vi 
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7. Laboratory procedures in industrial research. 

8. Simulabon of field projects in training for Peace Corps service. 

9. Student teaching in teacher education 

Simulator assessment typically involves a machine, piece of equipment, or 
material designed to simulate parts of an operational situation m which a person s 
performance Is to be evaluated According to Gagn6. the simulator is charac 
terized by three features. (1 ) it is representative of a real situation. ^2> it.is designed 
to provide specific controls over the situation, an 1 (3) it delibeiately omits some 
aspects of the operation that are variable or unpredictable.^ 

There are a great variety of simulators ranging from the simplest equipTnent, 
such as the artificial nfle for individual drill m rifle handling, to the complex systems 
simulator designed for use by whole teams of military personnel The relative 
degree of complexity depends on the task or operation being evaluated Gagne 
classifies such operations as follows. 

1. P/ocedures such as cockpit training for pilots, console drills to control 
missile firing, or procedures for calibrating electronic gear 

2. Mctor skills as in simulated gunnery practice or aircraft landing 

3. Identifications stressing immediate recognition of targets, types of 
airplanes, or terrain features 

4 Conceptual tasks related to equipment testing, .dvigation. and interpreta- 
tion of aenal photos 

5. Team functions as in complex war games or communications among crew 
members 

The assessment of piepared performana^s, another form of performance 
testing, stresses the evaluation of artistic renditions or demonstrations of physical 
prowess that are normdlly the result of repetitive rehearsing for a period prior to the 
evaluation Typical exainples include rehearsed performances in the performing 
arts— music, dance, and theater arts— cina m sports and phys.cal education ac 
tivities 

In some instances, the ' aiII be de^sed to evaluate purely individual efforts 
as in solo vocai and instrumental auditions. rensics. gymnastic exhibitions, track 
and field events, and physical fitness programs In others, the evaluation of indi 
vidual performance must be given m the context of group activities, a factor which 
poses many assessment difficulties such as the aoility to i:>oldte the performance 
and control the testing s.tuation Theie are suipnsingly few standardized tests m 
the performing arts fields because the ma^or thrust has been to aevelop measures 
of artistic aptitude, appreciation, and aesthetic attitude !n music, best e^a^^p^e 
of a test designed to assess both rehearsed and unrehearsed muSiC is the 
WatkinS'Famum Performance Suaie, a cstandaidized test for band instruments 
developed originally in 1 942 but still a significant contribution to muSic performance 
testing ^ 

On the other hand the field of physical education has given a great deal of 
attention to me Development of tests that evaluate the results of previous practice 
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Tests range from those measuring specific, discrete skills in individual sports and 
physical fitness to more gross measures of performing proficiency in either indi- 
vidual or group situations. A comprehensive, useful treatment of this complex area 
d measurement is found in the recent textbook of Franks and Deutsch, Evaluating 
Performance m Physical Education, which proposes a hierarchical model of physi 
cal performance and fitness as a basis for assessment, the major components of 
which are body compositio.., efficiency, endurance, skills, and sociopsychological 
influences. 

Unique Problems in Performance Testing 

All assessment opetdtes on the basis of a common body of principles concerning 
which psychometricians are in reasonably uniform agreement On the other hand, 
the problems that will arise m the application of these pnnciples will vary consider- 
ably in kind and degree depending on the ..ature of the assessment situation 
Thus, the utilization of expert judgment in the performance test poses different, 
though not necessarily more severe, assessment problems tt'^n it does, say, in the 
oral interview often used as a measure of subject-matter knowledge 

One of the most difficult and umqoe problems to be faced in assessing per 
formance is the question of its repre3entativeness and its relationship to the twin 
measurement demands of validity and reliability Although these concepts were 
discussed in Chapter II, the uniqueness of the problem requires a separate treat 
ment here. 

Simulation and the Validity Question. Earlier, it was pointed out that the pnnci 
pal characteristic of the performance test is its relative realism or simulation of a 
real-life situation. Furthermore, a b»mulation is cons.dered to be representative o^ a 
real situation when it is comprehensis^e, i.e , covers a wide range of aspects, and 
when It demonstrates /^c^e^iry, t,e > faithfully reproduces each aspect When these 
two conditions are adequately met in a performance test situation, one can ass^nie 
that the test has achieved a satisfactory degree of validity Stated in assessment 
terms, a performance test should produce a lOre valid result when the elements of 
the criterion situation are adequately sampled and realistically represented The 
failure to simulate adequately a real situation is one of the chief shortcomings of a 
great many applications of performance assessmont A case m point has been the 
common practice in teachei education of assessing the ability of college students 
to understand principles of teaching on tne basis of an observation of their teaching 
performance in a simulated ciassiuom environment where their peers pretend to 
be young children 

To illustrate more fuuy the need for adequate simulation to insure lest validity. 
*et us consider the case of a final examination administered at the close of the 
training period for a student preparing to become a stenographic reporter In this 
example, assume that the basic manipulative and transcnptive skills have previ 
ously been tested and that the objective of tt is examination to assess the ability 
of the student to perform satisfactorily m a bni. elated on the job situation, in this 
case representee by an audiotape of a courtroom scene 

To recapitulate from the previous discussion, the simulation must produce 
conditions that satisfactonly answer the following questions 



66 



( o 



1. Does the test create sufficient situational realism? 

2. Does It provide an adequate sampling of the various elements in a court 
room scene that are crucial to the performance of the '^tcrographer'? 

The following checklist of preparations is suggested to ^jrovide the simulation 
necessary to produce valid test results. 

1. Elements of the courtroom scene. 

a. Prepare two tapes of the scene, a first version for tryout with students 
and a second refined version. 

b. Set up a scene tnat utilizes a number of voices of d''*erent qualities, 
accents, and inflections 

c. Provide a combination of longer sm'^^ie speeches and rapid-fire verbal 
exchange involving several persons 

. d. Have the dialogues include a mix of declaratory statements, questions. 

short, long, and incomplete sentences, and interjections in the middle of 

statements and questions 
0. Create sulxlued interchanges, as between judge and counsel, and 

speech at high-volume levels 
f. Provide for backgroi^nd masking sounds such as spectator laughter. 

rustling of papers, and the like 

2. Equipment considerations 

a. Use recording and playback equipment of good fidelity, particularly for 
reproduction of the spoken voice which, to be intelligible, depends on 
recording and reproduction emphasizing higher frequency responses 

b Use loudspeakers, preferably two (* ont and back), rather than ear- 
phones to preserve the sense of sou. i coming from several directions 

c To preserve sound quality store the original tape and use copies for 
frequent repetitions 

d Make sure stenographic equipment is in good working ordei 

One method of handing the validity problem m simulations might be for the 
assessor to begin with a highly simulated testing model and. under successive 
tryouts olthe lest situation remove elements and note the effects For example m 
the illustration oresented abovo, it might oe feasible to create a videotape record 
ing of the seen? first, extract an audiotape version, and compare the test results 
achieved from both versions At the dose uf this process, the assessor could well 
find that for some assessment situations a less i.losely simulated model would 
sen/e to satisfy validity needs 

Relationship to Reliability. The vd... , , problem has direct beanng on the reliabtl 
ity problem and. .n fact, can pose a real dilemma for the assessor Real life condi 
tions, particularly where complex human interaction situations are concerned, are 
very difficult to control Now it is a sme qua non that sound measurement de 
mands control, the examinee should be evaluated each time under similar condi 
tions Only then can />e expect a test to be reliable, i e consistently produce the 
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same result over a period of time By increasing }t>e:»fepfesentaliveness of the 
simulation, we assume an increase in validity, ^t therein lies the dilemma. As 
validity increases, the possibility for control generally decreases and so. therefore, 
does reliability Without reliability, of course, validity has little meaning. Since both 
elements must be present, the assessor has no choice *^ut to create a reasonable 
balance between the two n siniUJs!"^9 ^ppropriaio conditions for performance 
testing, an achievement that cun be realized only alter expenmentation over an 
extended period of time. 

Another way of looking at the problem of reliability is to picture the evaluation 
of performance in educational settings as an assessment continuum. At one end 
we would be dealing with one on-one evaluation, one expert jujg.ng a single 
performance, and at the other large numbers of performance evaluations utilizing a 
standardized test, such as the Watkms Farnum Periormance J^ale designed to 
assess performance on band instruments. In the one-on-one Sftuation. iaue dis- 
agreement would be likely to emerge among experts when evaluating lower 
levels of performance, for example, the assessment of beginning ty^jing or short- 
hand In situations of this kind, the use of a single evaluator would probably be 
justified But as we move up to highei , more complex levels of performance where 
the variables increase, as m evaluating artistic renditions or performance in leader 
less group discussion, and the value onenteO judgments of assessors become a 
contending factor, the likelihuod of disagreement is much greater In such situa- 
tions, the ability to achieve an acreptable degree of reliability becomes more 
difficult when usmg the stngle expert At the other extreme, the use of the standard- 
ized performance tests m institutional settings can be completely unfeasible. Al 
though many such tests report good reliabilities, the low volume of students 
creates a very high cost factor In addition, this type of test is usually so inflexible 
as to be inappropriate for most performance situations 

A reasonable solution for the reliaoihty dilemma lies somewhere t>etween 
these two extremes The compromise normally would cal! for the use of multiple 
ratings, generally .nvolving the so-called jury system o' dbsessment The mure 
institutions become involved m competence baseo edu^dtion, ^ot\ expedience 
programs, Co.nmunity oriented activities programs and l ^ lecognition of lifelong 
learning accompiishmet.tb, the greater becomeb the need to develop performance 
assessments m the jury mode Although the cu&t may still be a factor for institutions 
witn small faculties, the trade off for improved assessment could be great In 
utilizing this assessmenl mode, the addition of a crucial element should be given 
serious consideration - the process -hould De supervised by institution-based 
psychometric specialists assigned amb of subject specialists functioning in 
fields where performance is demdi.ocd An example oi a model of this team 
approach to perforaiance dssessment is p;resenled Idter m th*s chapter (see p 71 1 

As an example of the ^alidity-reliablity problem, the evaluation of written 
performance in a foreign langu age presents ntertMing uialienges from the point 
of view of both test val. and reliability A hignly face valid technique m this 
regard would oe to have the student prepare one or mere real life documents such 
as memoranda, personal or business letters, tind so forth, and tu have these 
evaluated for overall quality Although the validity surh a testing situation would 
be s( if-evident, the rehabiMy with wh»ch the students performances could be 
scored might be tailed into question since globai appraisals of quality — 
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especially of fairly lenglhy,and largely unoirected texts— can produce wide varia- 
tions in the scores assigned, depending on the particular individuals doing the 
evaluations, the amount of preliminary training in the scoring techniques, the 
sequence in which the texts are read, and a number of other factors extraneous to 
the question of writing proficiency per se. 

To provide for a greater degree of reliability in the writing lest situation— 
especially when the papers must be graded mdividualiy by a number of different 
instmctors— One possible approach would be to break the student s writing task 
into a number of smaller and more highly controlled elements, each of which could 
be evaluated more precisely and more uniformly than would be the case with a 
longer and "freer text. Formats m which the student is asked to complete partial 
sentences by filling in appropnate verb or adjective forms, or to produce whole 
sentences which incorporate certain specified elements, usually permit easier and 
appreciably more reliable scoring. 

These and other ty^es of wnling tests involving student performance on a 
senes of discrete elements may, of course, be viewed as somewhat less genuine** 
naeasures of wntmg ability than 1$ the production of an entire business letter or 
other texts typical of real -life wnling av^livilies. If the former kinds of exercises are 
considered insufficiently representative of the types of writing tasks that the slu 
dent will, in fact, encounter outside of class» it may be necessary to adopt an 
intermediate approach such the Anting of short portions of oaca of several 
representative wntmg tasks (for example, tht9 opening paragraph 0' a business 
letter, with the top.c and purpose of the letter specified m advance) Th s procedure 
would have somewhat greater s luational realism than the discrete element task 
and would at the same time permit more reliable scoring than is the case with 
longer como.'ete themes 

Checklist of Considerations 

In concluding the treatment of this phase of asoessmer.i, the potential evaluator 
should be alerted to the special cond tions that ought to l^e ^^tisifed m developing 
and admmistenng performance assessments Coni.iderations specific to perfor 
mance assessmern can be derived fiom the moie general steps presenteo at the 
end of Chapter II 

The method ot presentation is a checklist of f. ve categories of key questions to 
be considered 

1 The porformance obfecVves 

a What aro the crucial behaviors involved m the performance'? Are 

some more important than others, and should they be weighted*? 
b Is sufficient information available or will detailed task analysis be 

necessary Example The more readily identifiable tasks of the typist 

as compared with the complex range of the functions of a supervisor 

of 25 file clerks, typists and stenographers* 
c Under what specific conditions are the behaviors elicited m real Uie^ 
d Are appropnate measurement instruments already available, thereby 

making it unnecessary to reinvent the wheel '> 
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2. 77je choice of assessment method » 

a. Can a real-Iifo situation be utilized, thereby avoiding the necessity of 
simulation? If so. can controls be set up to satisf/ reNab<lity needs? 

b. What is the cost factor to consider in determining whether to use a 
real situation? 

c. If simulation is necessary, what are the crucial aspects to be simu- 
lated? Is equipment adequate? Physical arrangements proper? Suffi- 
cient displays and aids? 

d. Is the assessment set up to enable close replicatpn for evaluating 
(Sfferenl students as well as the same student at different ^imes? 

e. Which IS more appropnate, product or performance Assessment, or 
both? 

3. The assessment process 

a. Have appropriate criteria been set up to choose assessors properly? 

b. Have adequate plans t>een made for proper obser/a^ion and record- 
ing of performance? Example: Determination of the appropriate 
means of recording among tape recorder, videotape recorder, oi* 
movie camera. • 

c. Have necessary sconng procedures and forms been set up, sucfi as 
checklists, rating scales, etc.? 

d. Have absolute and relative weights of assessment elements been 
established? Sramp/e. Evaluating ihe pilot in handling essential land* 
iog procedures versus the form and sn}oothness of the landing. 

6. Can one expeit do tha evaluating, or are multiple raters preferable? 
Example: Generally suffiaent to use single rater in evaluating lower 
level skills, advanced levels require several raters. , 

f> What training procedures have been developed lor assessors? 

g. Have sconng standards been checked periodically when tests are 
repeated over a period of time? 

4. Assessment administration 

a. Have instructtons for the student been developed? 

b. Have- necessary assessment controls beuM established, including 
desirable physical conditions^ sequence and timing of activities, and 
the like? 

c. Has test secunty been worked out. such as «. > development of suffh 
cient aiternite lists of tasks from which to choose? 

5. Economic considerations 

a. Are the results of the test of such impot tanco that a high cost may well 
be justified? Example The necessity of evaluating a real perfor- 
mance of the potential conductor of a professional symphony or- 
chestra a$ compared with accepting the simulated testing of the stu- 
dent who must satisfy conducting requirements for the Bachelor of 
"Music degree 

b< How long is the assessment or test expected to be viable? 

(1) >WiIi the volume be such that developmental costs can be re- 
couped over a period ot time*^ 

(2) Is the field changing so fast that the particular test will be out- 
moded? 
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c. V^al are the future needs for modification of the assessment sys- 
tem? That is, how much flexibility will be necessary regarding perfor- 
mance content or tasks and standards? 



A PLAN FOR THE USE OF EXPERT JUDGMENT IN PERFORMANCE 
ASSESSMENT 

This section presents an example of a procedural plan for performance-based 
assessment of experiential leaming. It //as developed at an external degree institu- 
tion that relies for the development and administration of Its assessn^ent proce- 
dures on a close working relationship between the institution's professional staff 
and a group of approximately 300 outside academic specialists. The plan is viewed 
as a procedural goal, although most of its components are already in use. Other 
institutions may find it useiut as a model of one possible approach to performance 
assessment. The descnption of the plan is followed by representative illustrations 
of pfjrformance assessment. 

Selection and Orientation of Performance Assessors 

Assessment staff at the college or university solicit recommendations of qualified 
assessors from academic deans and other college or non-college sources Impor 
tant cntena are. subject-matter expertise, an appreciation of the operation and 
philosophy of external degree programs, and a willingness to become familiar with 
performance-based techniques for measuring student accomplishment Vitas of 
proposed assessors, lists of courses taught, and other documents are reviewed a? 
part of the selection process, and wherever possible an informal interview with the 
prospective assessor <s earned out by college staff. An ongoing informational file is 
maintained for each participating assessor showing the assessment projects in 
which he or she has been involved and related comments of the assessment staM 
of the institution. 

Each newly-appointed assessor receives a package of explanatory materials 
descnbing the organization ano philosopr^ of the college and the basic procedural 
steps to be followed m the assessment program. (A loose leaf handbook, which 
can be added to and revised on the basiS of continuing experience, is a useful aid ) 

Following their review of background malenals. asse-:oors attend a joint dis 
cussion meeting with the co ege assessment staff to recuive more specific infor- 
mation on the particular type of project in which they will be involved and to discuss 
the general outline of the assessment plan. To the extent possible, all assessors 
for a given performance area are asked to be esent simultaneously, so that all 
participants receive the same type of bnefing and orientation to the project 

Development and Tryout of Assessment Procedures 

in situations involving on-site performance evaluations, assessors and college 
staff make an exploratory viSit to determine that the physical layout and "am- 
' biance of the site, including me presence and condition of necessary equipment 
or other support elements, pemiit a highly valid, real life" demonstration of the 
performance to be assessed. - 
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One or nrwre assessors* aided Jby college staff as necessary, review published 
instruments or dccurnented procedures in the area to determine if appropriate 
measurement tools are available or can be readlly^adapted from existing sources. 
Any findings are reported to the full group of assessors for discussion. 

Assessors and college staff draw up a formal plan for eliating, observing, and 
evaluating the student performance. T^e primary considerations in the plan are, 

a- Providing for an adequate and representative sampling of the performance 
to be assessed, 

b. Presenting directions to the student that adequatel*' onent and prepare him 
or her for the task, Ihese can range from a short list of procedures to 
well-developed study guides, 

c. Developing an observation system, including any necessary forms, that 
can be eriiciently and accurately used m the actual assessr^ent situation. 

Assessors and college staff, working in close collaboration, develop neces- 
sary procedures and instruments to implement the evaluation plan, A variety of 
approaches are used, depending on the number of assessors working on a given 
project and the relative difficulty of concepjtualixing an^or developing appropriate 
assessment techniques. In very straightfonvard subject fields, whore all assessors 
are quite clear about and in agreement on the proper evaluation techniques, indi- 
vidual assessors may be assigned to work up different specified portions of the 
entire assessment procedure, which ate then combined, often with^hght revisions 
foltowing group discussion. In more complex situations, the ?ssessors may work 
as a group, with a high level of input by the college assessment sfeff, and may also 
submit their work for critique and suggestions by an addition^ consultant in the 
specialization, ^ 

Simultaneously with the development of procedures and instruments, the as- 
sessors and college staff discuss the cnteria and standard/ to be used in judging 
the student peiformance. Depending on the nature of the (Teld, the student may be 
required to meet certain minimum performance standardi m each of several sub- 
areas, or superior performance in one ^ispect of the assessment may iegitimately 
be considered to compensate for less successful performance m other aspects. 
Related decisions are made on whether a simple credit no-credit evaluation le^rt 
will be made or whether several different levels of accomplishment will be identifiec 
and reported. Additionally, a decision js made on whether supplementary narrative 
desaiptions of observed strengths and weaknesses jn the students performance 
will be provided. If necessary, assessment piano and draft instruments are revised 
to meet the information-gathering and reporting goafs- decided on. 

In situations where the volume of assessments warrants it, a preliminary Iryoul 
pf 'the assessmsnt procedures will normally be conducted, in A.Mch each of the 
assessors for that project simultaneously evaluates a number of student perfoi- 
mances by using the draft procedues developed. Results are reviewed in terms of 
the efficiency and straightforwardness of the rating procedure and the extent to 
which the assessors are able to give closet;, . mlar ratings oi any given student. 
Problems in either of these two areas indicate the notid to discuss and revise the 
assessment procedures. 
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w ^ ?^^,tlonal ApptlcaUon pt AssessfMnt Procedures 

. . Audonf s performancnis ev^ualod^sirmiii^eously by ai masl two assessors 
^vklod for In the asse^ment^plan, {t\e student has been lulfy alerted and 
spired" for the assessment process aijcf ts given tha-Oppoftunity lo present the 
total range^ of ^behaviors under constderat'on, In sic /me i/>staoc«s. th^ ffes^4ires 
observation of the students psdomance o/^ two oi nrwra separate o<xas*pns Or 
each occasion, the student ts granted a warm sup penoc lo become acc^mab^ed to 
the sihjation. 1nckKfing^ny.equipn>ent or apparatus. 

^though the assessors rfeseni simuttaneousty.at the performanceiSi 

the Student, rating notations are made mdependenuy for later discussion and 
" reliability checking, (Note, in some performance assessrrjent situations, *s impos 

SWe for finandal or other reasons to use more than a singte assessoi pei stvident 
Wiese instances, mttiai grotJp dovejopment of the assessment proceciure and gnsup 
determination that the raung tectinjquo does .perrrfcgf roiiabto scof^ng b/ a single 
, individual offers some justiRcauon lo? the stng^e-^ssessor appioadh- In ongoing 
] programs that must use a smgie-assossor tecfmique. penodit spot che<*K5. In 

one cwr more ad<Sliona) assessors are v^iic^d tn to y>ve .ndependcnl latinofs of the 
« student's penw ^rsnce for corr^par?^* purpoites* <iie s^arned ^uX <frhefe%*a» po^Sr 

ble.) 

Scdring and Reporting of Student Performance 

When more Uvin one ratei ns iShoC tn© evaiwalton o! v/^j^ rvutipJe (MKfOi 
minces, tne assess^a ana iAwiege stall meet ai> a 91 to review and uiscu^ *M 
5^ stu{^m pertoonange tcjfefii^s Tabu^atsons a! Ih^ .atngs a?>d related t;l^^^;ltca: 
descfip4K)ns ar$ presented ar^S interpreted ths assosSM? ^ t$ ^^ies^ ^ 
Average ratings of aii assessors ewatual^ng a g.ver. student M>ns14^'1ne 
official rating, except wt^en an obvtoui.i> aberrant /^T»g ^ c^saDofUetTthfOi^gf 
* group dJSCussK?n and agreemem hia£4al^«Na JeisU^ptvJr^s A^f perfgi^^ii^Ke. .n^^u^r^j 

any applicabie^ suggestions (u me ';?!udenl jegatJin^ rjeedod ad^^lortat wo^k 0^ 
^ V preparation, are devejopeo oj* a yfoup uai*^:* and .T^acJe part u? tho ^'na' ?epc^ t^s 
student- 
Project Review by College Staff and Assessors 

A portion of thp scoru^g ana reporting 4^ voted lo a 4-r*t^vai 0' th6 (^im 

assessment project, inc^ud^ng sug^^e^in^ir^ ^on<»ept«a'^nd pfucedu%^^ .mprovQ- 
ments that couJd oe made <n future repeM^uns ut ihe proves^ e Jt^a n 
subject»matter area or in mo college s as^^mtr? i po^gj ar** il^^^f aUj i'' to^a. st-af 
review 0! eacn coriipieted asses 2^e.u p^y,^,* a vA*t f£>T ft>e sojt^o 

revision and development purposes 
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Coinmmn Assawt^em s^s^pJd »t«on>5^a!o stanch 

* 0rys*<^og5caJ.t>as«3 of adc^c^ 

. & mg^X kmn^ ot m<rtvldual« A^ady HoWJng FAA {FodordI AvUUon 

f AA patnai o^pcm thai i^r»sufo fOQu^emonis 

offrfnendasoos d A£RO iA/»aim Eckx^^bon Review Oraamzaton] Co^ 
ki^ Cm^i Standard A&^smem a^a^Jal^ to ^nd&^uals *io^i^ 
rat?>^ 10/ J^^mls F^^ol. C&nr^merofii P^^. mstf^m^i P^-t f^^^l lat^if adof 

e. l*«Sajl UbofMory Teclinoiogy 

Ca^m(*^5 K;5ow*^<j 0^ iait>^at<jr,f (0«^r^uos may o< ma-jf not be 
c^t>i<5s3 vi^ir* oc^do^^ K.n*:?fr;0^a rented m^^ S^Mi can c?© 

fr^a e^ectiv^ v»t^?s*^Pf:^e^;^^a^B Cc- n^^^ t^cnOo^^ u^^ft'^^Q 
,s f^i a C'^ef0Qv<^ ^ts ^'^'^ a£* p^^ijrrr. a <: ^ set'^'v^ 

7 f>hy«i» 

Kf^se^s^ ^^-is (^©/rp^'e^v::^^ ar-i ^enV ^J^-^^'W Jurist J^/^c^io^Ss^ at m 
'^itr^-^ t "^c is^c^^r. 0^ ^T<t\ fcr*s;^'^:S5^ covert Ir/ tn© 




8. Homo Economics — Food Preparation , 
Speaat Assessment Techniques 

• Definjltoo of bfoad-scale problem that will involve the evaluation i 
nutntfon and other dietetic factors related to meat preparation 

• Oosigndtk>n of a specific situation that will call on student to dorn- 
onsirate a vanety or compt>tences and the knowledge rt ^uisite to a 
satisfactory outcome 

• PreparaUon of a "shopping fjst" that will adequately cover the needs 
stat«Kicbove ' 

• Evaluation of items purchased 

• OoservaUon m a kitchen setting of actual meal preparation 

• SampSng of outcome and evaluation of final product 

Comments. Vanous slated cntena will have lo be met. These would be the 
standard agreed-on criteria inat shouid be accepted by anyone m the Ueid. 
The particular method allows for the statement of a prubiem whose solution 
calls on a vanety of skiHs at different levels and satisfies the demands of 
different parts of tho curriculum One or more of the following examples of 
parts of the curriculum can be stressed volume feeding, gourmet cooking, 
cookiag tor the invalid, nutnlion. consumer educatiorn product development 
m a commercial setting, 

9. Foreign Language Speaking Proflclency 
Speaaf Assessment Tezhniques , 

• Face«to-face interview of 20-25 minute duration 

o Student proficiency evaluated m terms of performance capabilities ^n 
reai-lifo Situations 

• Rating scale encompasses entire range of spsakmg profioency 
from survival level to proficiency shown by educated native 

» spe<xyets 

Comments This language proficiency interview «s based on techniques 
onginalJy dovetoped by the U S Foreign Service Institute and recently 
adapted for use in Peace Corps language tramtng. teacher cert/f/cation 
ptograms, and othei situations The ^nterv aw is conducteo by eaher one o* 
two trained raters who converse with the examinee <n a 'oiativeiy mformai 
settirjg on topics representative of a number of roa^-life communfcation areas 
Throughout the interview, the pace and level of soph'Stication of the discus- 
sion are increased to a point at which the maximum ievei of performance has 
been attained An 1 1 pomt scale ^5 baSiC performance levels, together with 
«ntefme<fiate plas values) iS used to rate the total performance Each of 
these levels \z defined by a short paragraph descnbing the kinds, of 
language^ use situations m which the oxam.nee is considered capable of 
communicating m an effective and apprupnate manner 

Concluding Note / 

Several of the above inu$trat.or,i, contain elements v. ^roduu as&o&j>ment, namely 
numbers 3, 7, and 8 However, it should be »tre .^O tfiat the principal focus of the 
assessment is upon tf^e perfofmance of the .odent a situabon where the pro- 
duct IS introdiiced as a means of identify*. if.e performance skiHs necessary to 
create the product 
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Assessment of Written Material 



^Myrqa Miller 

The outcomes of learning that has t>eeri gained Ihruugh eApenence in a nonulass 
room se**jng are aK)St oUen communicated via the written mode, and experts are 
called ^.1 to measure and evaluate this wntten work. The form of the work iS usually 
one of the following, an essay examination answer, an assigned topic essay, a 
general essay withm the portfolio of prior experience, or a log, diary, or journal. In 
addition to the general concepts of reliability and validity, experts making academic 
judgments about wntten matenals need to pay particular attention to the pitfalls 
implicit in three significant questions. Is the written work ad^/ecf sample of the 
learning clairr.ed? (2) Is (he written worK an assertion that learning hat occuired m 
the past or in a distant setting? ^3) Is the written work an indirect scimple from whicr 
the expert can deduce cognitive and affective learning? 

Of these three types, the direct sample is moit like the product of classroom 
learning. In general the assessment of ihis type of written material requires little 
modification for application to experientiaily acquired learning. Because of the 
similarities to other product assessment, be sure to consult Chapter IV for the 
discussion of important issues in product assessment. 

Tlie direct sample written product is often an essay examination answer, an 
assigned topic essay, or research report if the sti- jent claims to have learned 
American history through traveler self directed readuiy. he c^n be given an essay 
examination by an expert r*.otorian. The historian can measure and evaluate the 
student s knowledge by u^^mg trie appropriate cnter*a for what constitutes a correct 
response. In aTiotner case. .! the student i>eeks credit for English composition or 
communication sKiil coripelence, an as^igued topic essay can be evaluated as 
direqt evidence of writtp>n verbal abil.ty and Knowledge/ A third direct sample might 
be a research paper submitted m Spa.i.ofi for a student claiming to have learned 
Spanish while living abroad. Such di.t;ct oampies are clearly useful for a conipe 
tence currently heid and easily demonstrated through summatwe evaluation. 

All direct sample wntten responses require ex^ ^it agreed upon cntena for 
tjvaluation. wnich have been made avduabie to both the assessor and the student 
before the actual evaiuahon occurs In the classrooai each student gradually and 
subtly comes to Know :lie professor s expectaticns. Although the professor m<sy 
not have made known his standards at the Ouw^t of class, experienced students 
usually learn to mler ihe criteria for evaluation and modify their study plans accord 
ingly. This ability to discern evalciative cnterit. m^y. »i. fact, be the actual noasured 
skill of "successful : students. 

Continueu classroom intera^.v,^ii throughout the semester gives classroom 
students the opportunity to son out information relevant to expectattuno for mea 
surement and evaluation In addition, the experience with mid course formative 
assessment processes prepares studentj» through tnai and error learning for final 
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summative evaluation. Usually the student whose learning has been expenentially 
derived cannot reap these benefits of prior knowledge of the professor or formative 
assessment. Generally, the teaming outcomes of experience are judged by an 
unfamiliar expert. Whenever this is the case, every effort should be made to clarify 
what IS expected, to make knowr what cntena for evaluation are to be applied, and 
to describe all other •'rules" for assessment. 

Students in on-goinq experiential settings often keep logs, diaries, or journals 
of their daily learning activities. These written responses are then submitted to 
experts for assessment of learning. Thr^jughout suqh journals, there are assertions 
that learning is taking place such as. I am finally beginning to enjoy typing now 
ihat I am only correcting one error per purchase order. Acceptance of such an 
assertion should not be substituted for an actual typing test or a letter of verification 
from a supervise. This rather obvious example is given m order to contrast it with 
the more common and more difficult cases that frequently arise. For example, a 
student working on a psychiatnc ward nhay claim to oe learning abnormal psychol 
ogy through his daily observation of patients. Another exar^ple might be when a 
student working in a nursery school describes the children s daily activities and 
claims to be learning early childhood development. In these cases, the task of the 
experX is to distinguish beiyveen those statements that are mere leporting of obser 
vations and those that indtuate thoughtful analyses based on college level concep 
tual frameworks. _ 

This issue of assertion versus true learning outcomes is crucial m the assess 
ment of experiential learning. The distinction lo, of course, often extremely difficult 
to make. Most of us are unwittingly committed to the belief that seeing ts leaming 
(on some level). Our children may say, I learned a lot today. I went to the fire 
station and saw the fire engines, or a friend may report. I went to India and saw 
the faj Mahal. In truth, these €ttmulc.ang viSua! experiences plus the personal 
experiences accompanying them, so often do iead to learning that we make the 
incorrect assumption that alt such expenences lead tu significant Jdammg for a'l 
learners. Experts need to keep m mmd that the experience, setting, or acf/vify iS 
not the learning, it is the opportunity for learning to occur. Therefore, in making 
judgments, assessors must carehlly sort out the students descriptions of learn 
ing opportunities from ai^tuai college ,evel Jeaining deservtng the a^ard of credit 

The portfolio of prior learning is especially susceptible to this problem of 
distingufshing opportunity from tear ^ng. It is difficult for even the most expeuenced 
assessor to sort out actud. -^arnmc from very persua. •^e descriptions of learning 
Talented students with exce^^tiona ?riting ability car. . .en provide very convincing 
cases-for credit. The expert may lj crediting writing e/.pertise when called on to 
evaluate other swlis. Or he or she may assume a connection between the wndng 
ability and the learning outcome claimed i^>ee halo effect. Chapter II) it is often 
even more difficult for the stuoent to sort out.hi&her own lea^^tng from experience, 
and portfolio preparatiori almost always iei^utieb that students disentangle leammg 
from experience.^ Most prograrhs have <ouna ;^at a strong counseling component 
is particularly necessary for aider students who have been away from the 
academic world for some time. 




Secause pasi learning is a unique personal experience, many experts may 
feel that student assertions must be accepted. After all, the student is the verifier 
that he or she was there at the time. This does not mean, however, that the 
students' assertions are synonymous wi'h leaming outcomes. Again a good coun- 
seling system can help students to develop self assessment skills that can provide 
reliable statements rf learning outcomes. In this way, not only the 'acuity assessor 
benefits by having clear stateme; is of learning outcomes as the foci of assess- 
ment, but the students also.gain a new usable skin,^ 

Although it is essential to separate mere assertion from actual i(^arning out 
comes, the separation of personal experience from individually internalized leam 
ing IS somewhat artificial. Human lives, experience, fee^ngs, thoughts and learn- 
mg, are in fact, all of one piece. Students oftentimes respond emotionally to judg 
merits of past learning activities and frequently perceive these judgments as 
evaluations of the worth of their lives. According to Richard G. Beery, in his article 
'Fear of Failure in the Student Experience, our society reinforces the idea that the 
measurement of achievement is equal to the measurement of ability, whfch is 
iiiterpreted by many as the worth; of ones life. For this reason, it is highly recom- 
mended that experts evaluating pnor teaming always be disinterested parties free 
from constraints in dist.ngu(Shing learning from described activities that migKt be 
construed as the total lifers worth. 

The written response, whether essay exam, assigned essay, portfolio essay, 
or pumai, can often provide a sample, of the students cognitive and (less fre- 
qut/nclyj affective processes from which an expert can deduce creditable learning 
outcomes. If we select from Bloom s Taxonomy/ we can choose specific cognitive 
processes to be useo as cmena .or judging written matenals. An example might be 
a Gtudent who was a Peace Corps volunteer and who describes his experience 
with the problem of offsetting mainutntton in a remote South American village. He 
might indicate a comprehensive Anurv.euye of the situation by providing esse^.l.al 
data on eating habits, agricultural practices, birth rates, death rates, etc. He might 
demonstrate his ability to analyze the problem through discussion of the factors he 
chose to assign to higher and lower priority. Synthests might have been dem 
onstrated through the combination of resources he used to address the problem. 
The ability to evaluate might have been shown m his choice J priority to attack. 
I.e., encouraged change in eating habits when iacK of new eM^ipment made a 
change In agricultural practices unfeasible. 

!n a Similar manner, interpersonal skills and understanding of others might be 
assessed by reading this student s descnpt of his personal adjuOtrr^cnt to the 
mores, customs, and roie expectations oi ew culture. The expert should not 
base his judgments on a statement SUCi ' •mo t grew and changed a lot, or I really 
learned to understand the people IwvurKed //ith The evaluation, although based 
on the student s descnptior.^ of his parti .ipation in the delivery of a baby, should be 
derived fiom thoughtful reflection on ihis experience. If he was able to provide 
help within the context of a second culture, and can provide details of his self- 
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avVareness and understanding through discussion of his personal attitudes (cultur- 
ally derived) toward the mother, the methods of delivery, and the infant, then credit 
for these skills may be quite appropriate. When alter7;ptmg judgment^ of cogniti ie 
and affective processes, experts musl be careful to avoid the.pitfalls of creditin;! 
assertions, and, In this case, not award credit for the biology of child-birth," whic^k 
has not, in fact\ been demonstrated. Or, if ou ^Peace Corps volunteer happen^ t j 
have oOlSicrJino writing ability, the experts m jst ^ivoid the pitfall of crediting verbal 
facility when called on to ^udge analytic, evaluative, or interpersonal ability. 



CATEGORIES OF WRITTEN MATEfilAL OFTEN CONSIDERED FOR 



Much of the research literature has been devoted to contrasting essa^ 
examinations wi'th'objective examinations. In assessing experiential learning, 
basic assumption is that objective exarhs would be too costly to produce for ^le 
limited number of students who have had the same or nearly .similar experiential 
learning. Those students who have acquired either broad .earning or highly 
specific learning similar to traditional course worK '^an take the already available 
CPEH or CLEP exams. The concern of this Handbook is with the assessment of 
unique 'experiential learning both m sponsored settir j and prior learning. For 
these pui poses the essay may be. appropriate. 

Experts who are asked to compose essay questions or to rate essay answers 
should be provided a common definition of thts type of assessment task. A df 
finilion which includes the weaknesses of the essay and v^hich indicates posstLie 
problematic areas is provided by ?*alnaker. 

... the essay quesbon is defined as a test item whtch requires a response^ 
composed by the exammee. usually m the form of one br more sentences, 
of a nature that no single response or oaltern of responses can be hsfed as 
correct, and the accuracy and quality of which can be judged subjectively 
only by one skilled or informed m the subject The most^sigmhcant features 
of the essay question are the freedom allowed iho examinee and the fact 
thai not only can no smgte answer be hsted as correct and comploio, and 
given to clerks to check, but even an expert cannot usually classify a ra* 
sponse as categoncally nght or wrong. Rather, there are different degrees 
of quality or merit which can be recogn»zed/' 

When exports are expected to make judgments on degree of qu<^'*y, they 
musl^be provided with clear crilena Most writers su^jge^t that essays sh\,uld be 
read qutekl^ for an over all impression, especially vvhen they are being judyed foi 
quality of writing as ^.ell as for substance and when ^.lere iS a relatively iargt> 
sample of papers on the same topic. Readers should be asKed n^t to analyze 
specific points or to note trivial weaknesses. A holistic or global approach is 
get>3rally favored. Even *f there is only one essay on a gwen topic, A should be 
reac by at I^^ast two judges whenever possible 
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The tasks should be clearly defined so that the student can be made aware of 
vA)a\ is expected. AlLquestions should be carefully worded so )hat the student 
understands what he is expected to do. If an answer of a particular length is 
expected, specify how long it is to be. If the student is allowed to expand upon the 
question, state this. If he is expected to stay within given limits, make the bound- 
aries known to him. When the students are expected to set aside mechanics and 
wording and to concentrate on ideas, form, and flavor, they should be aware of 
exactly what they are being graded on, and readers should be selected who agree 
to grade these factors. Sometimes, of course, readers who agree to grade on 
these factors may nevertheless find themselves influenced from time to time by 
grammar, spelling, etc. Students therefore should be torewamed that it may be 
impossible for evaluators to totally ignore mechanics and wording and that errors m 
grammar, spelling, etc., may affect the evaluation. 

Portfolfos 

The portfolio of prior learning usually contains as one portion a general es^ay. This 
essay is frequently assessed by a faculty expert or panel of experts. The essay is 
often autobiographical and highly descriptive of past experiences. The essays can 
range from a few pagos to full booLsize manuscripts. Details may include 
emotional reactions to past events and intrapersonat insights as well as job de- 
scriptions and catalogs of skills attained. With different portfolios, the experts who 
assess the learning in the portfolio may do so from different perspectives and with 
different criteria for assessment. For any one puitfoiio, however, Cie criteria should 
remain the same across experts. 

1. Descriptions of Past Learning as a Measurement of Wilting Corr.petence. 
Most institutions of higher education have as a stated requirement for the degree 
an acceptable le.el of writing competence, ThiS competence is frequently met 
through the preparation of the portfolio essay. Elxpds who are called on to assess 
the portfolio essay should be provided with a sin^ie set of crtieria specific to th'u» 
purpose such as: 

a. Clanty of presentation, proper organization, appropriateness of style. 

b. Logic and order of pref.entatton, adequacy of transitions, unity, and 
coherence, 

c. Neatness, lack of typographical or spelling errors. 

d. Accuracy of punctuation. 

e. Correctness of grammar, adequacy of vocabulary 

The facjity of each institution needs to define and establist, critena appro- 
priate to the student body and consonarU witn the phnosophy and standards of that 
institution. 

2. Articulation with Degree Program* The portfolio essay frequently serves as 
the vehicle of articulation of past iearoing with .earning to be undertaken at the 
institution. The task, in this case, iS to assess the student s ability to analyze the 
significant relevant learning from tne past and to integrate thio learning into a fiill 
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degree program. Depending on the particular otjjeclives or goals of the inslilulion, 
the criteria forihis assessment will differ. lE^pSrfs will require a S6i of clear cntena 
when they are called upon to evaluate the complex processes involved m dredging 
up, sorting out, categorizing, and articulating past learning. An example might be a 
student who decides to study oceanography after many years of worK»na as an 
engineer. His work experience might provide him with the knowledge of 
gci^hysics and other course work he needs. Much of his earlier academic work 
rhay njbw be rusty or out-of-date and may need to be discarded. Some of his Navy 
training may be, after close examination, highly relevant. Also his avocations of 
sailing, boatbuilding, and marine natural history may fit in well with a study plan that 
calls for an at-sea internship. 

Criteria for the assessment of this type of essay might include. 

a. Ability to present a well-organized logical argument, ; 

b. Ability tc identify pertinent leaming from expenence. 

c. Ability^to make decisions with well-supported defenses. 

d. .Ability to justify and to substantiate each decision. 



Logs, Diaries, Journals 

In the traditional classroom, the faculty member responsible normally has control 
over the matenai presented, the quaiity of work to be undertaken, artd the quality of 
the readings or othtr matenals tc be used. Since the subject matter emanates from 
the mstructorin the form of assignments or lectures, there is little doubt alx)ut what 
was taught— only about what was leamed. This controlled teaching process allows 
eacti faculty member to devise measurement instruments based on what was 
dativorod and, hopefully, received. In the expenenttal sbttmg, however, much of 
this teacher control is cjiven over to nonacademic supervisors, employers, or the 
student. As a consequence, one rather common method of recording the learning 
activities has been the log, diary, or journal. The student submits this log to an 
expert for evaluation feither as mtenm or tmai documentation o( learning. When a 
log IS used without on-site observation, the e^^ert is called on to evaluate bot 
what was delivered ^avauable iearntng expef^ericei and what was r^eived 
(learning outcomes). 

1. Description of Learning Activities. Often the log. diary, or journal takes the 
form of a direct daily record of each learning activity. Studen'*> m internships. 
on-the-joD teaming, or apprenticeships may .nctude a fine level of jetail for each 
new task undertaken, level of responi>ib»j»{y accepted, or skill ^.eveloped An 
example might be a teacher 6 aide who iot a&signed cie child vith a reading 
problem to work with and desaibes for eac^* day the phooic dnils. word games, 
and stones used as part of the student s teav^ier learner experience An bxpert 
reading such a jouinal could assess the leariMPo by uoing cnten;^ such as 

a. Ability to apply theory to actual situation. 

b. Ability tc interrelate facts, pnnciDles. and pht,non *?n^^ m a new jvay 

c. Ability to demonstrate the development of a ntw ^f.t of interrelated coi) 
cepts. 
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2. Ret lectloDj t^ognltlon, Affect The diary or journal can also be bsed to record 
the students' thoughts about and feelings toward their exponential learning. It is 
often this very reflection upon the expenence that produces the toie learning 
outcomes. A student of drama may gain more insight into her art after considerable 
reflection than on-stage during the actual performance. A social work intern, after 
many hours Of informal sclf-analysis, may recognize that she has been too quick to 
make decisions for her clients. Students of wnting and literature may use the 
journal to record books read and reactions to the books, as well as personal 
creative efforts such as poems or short stories. 
Criteria for the reflective journal might include; 

a. Ability to draw anaiogi.-is between related expenences. 

b. Ability to integrate experience into larger panems of meaning. 

c. At;!!ty to demonstrate that the experience shaped the student s views or 
philosophy of life. 

(See GAEL Institutional Report No. 3, University of California at Los Angeles, 
for further discussion of the L-se of the journat for spunsored expenenliai leaming),^ 



GENERAL CONSIDERATIONS 

important steps that should be taker* into acujunt ^hen dbije^sing written matt>- 
rials Include the following: 

1. Determine the specific factors on which the student is to be 
assessed Mote making any judgments. Make sure that \he student 
has a clear understanding of exactly wL^jt factors (e.g., wntmg com- 
petence, substantive knowledge in some area, etc ) fife lo bo 
assessed. 

2. Obtain a represent JiVe^sample of the students wnting , r 

a. If the assessment T^-ta be Jbased' on prior written j^^rH. provide 
guidelines to the", slydefU so U^al a reprosentalve samp!o of 
writing related \o the qhtaria can be assessed If actudent claims 
to htive :written 20 short stwes. for example, the assessment 
anouid be based on more than one or two of the stones 

b if the assessment is to^^ teased on a requested writing sample. 
^ the directions and guidelines to the student shoukJ bo clear and 
relevant' lo the factots established 

3. Prepare ?nr assessment 

a Through committee action or other methodi*. standards should 

be established For oxarDple, a file containing samples of 

acceptable levels of student work can be set up 
b C(ea>' guideHnis shoukJ be established for judges so that they 

know exactly what factors .'^e to be assessed, 
c Through workshops and or practice sessions, judges should bo 

trained to use agreed-upon procedures and standards 
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4. Make final assessment report 

a. Clear guidelines should also be provided for me wntng of a final 
assessment report. If the report is to bo of diagnostjc value, for 
examprt% the assessors should have some predefined structure 
around which such a report can N» wntten The report shouic* 
clearly relate the fsnat assessment to the wntton matenal 

Whenever possible have at least (wo ludges read the rnatonai 
Each judge should not know the other s evaluation pr»or to doing 
his or her assessment 
c As a rule, judges who are personally familiar w»th the wnter 
showd noi be used. Wtienever possible, the student aUhor of 
the materials Siiould rema n un^dentifioa 
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,;the' prfeecl!rig:chaptei^ ha^ each Resented some Important considerations for 
a$M9^^ six basic steps^hlch were recomniended In Chapter II reoccur, 
on various pointSi In the chapters deajipg with specific 
.mialh^ d a«9Ne^hrmn^^ it may be helpful to review these steps as they apply to 

. fiMac^^ of iwessrnept* Rgore 5. "A Matrix of ^sessment Steps/* presents In a 
the specific adapiatloa of each general step as It pertains to each of 
foupm^ asse[ssmerjt4echniques discussed In this Handbook. 
Rgure 5 cfeariy illustrates the similarity in basic principles that should be 
appBed to any assessment. The first major step. Establish Criterion Stj^ndai dSf is a 
9Qpd exarhple* the id^a is essentially the same for ail four types of ^ssessment 
rf*aft6cte» ;d^^^ e)camples or descriptions should be used to illustrate what is 
ni^ant 6y different levels of the criterion. If performance is Joeing assessed, this 
might' be xibne w|th/a number of direct examples of performance. If products are 

rbalpg^assessiiid; acUiai examples of products would be t>est, or if that is not 
pp^ibie/ clear veital descriptions of products could be used. The second basic 
stejd Includes two/dctlvitles. First, within each of the four assessment techniques, 
fiere is a.range of possible approaches. Careful consideration of the, assessment 
problem at hand will usually reveal one or two approaches best suited 1^ the 
situailc^. If interviewing IS used, this may mean choosing in oral testfrather than a 
semistructured Interview, foe example the second activity in this step involves 
providing some structure fori the assessee and assessor. Structure is what distin* 

.^Ish4s.an iniervlow from a casual conversation at a^ cocktail party or product 
'assessment from a visit to ah art show. 

- the. next step. Planning, includes both the preparation of Judges and. the 
administrative details, such as providing an adequate room. An environment free 
of distractiQg^noise Is cieariy essential for effectiveM'nterviewJng, and for many 
types of performance assessments special equipment will be needed. 

The fourth step has to do with directing and focusing the assessor's attention 
to only what fs observed within the structure of the a^essment probedure. The age 
or appearancaof the artist should not rn any way influence the judgment of aisses* 
sors considering artistfc products as teaming outcohies, for instapce. Once judge- 
ments have been made, it is important that documentation of tlie assessment 
process be ayaiiabte for review and that feedback be given to the, student, this fifth 
step may vary considerably in terms of the nruxte of transcrip^on. Products* for 
exarhple, serve as a record themselves, whereasjnterviews must be electronically 
recorded or detailed with written notes. ; 
' The sixth and final step can take a variety o^fonfns, ranging from judgmentally 
evaluatlnrg the adequacy of an assessment procedure to a fullblown empirical 
research study. Monitoring the quality of assessment Is a cr(Jfclal> but often ne- 

-g^ted* activity. Within a given institution It Is often difficult orfmpossibfeto apply 
the more.powerfuJ empirical procedures to evafuating^assessn^ent simply bjBcause 
not enough students are assessed with a given procedure, IRmpirical procedures 
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Figure. 5 
A Matrix of Assessment Steps 



Basic Sieff^"^ 
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Assessment 



Plan the 
AdtTvnistration 
of Assessment 



Relate 
Judgments 
to Obsetvations 

Record and 
Report 



//onitpr 
Assessment 
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Interviews 



Should be given in 
behavbral and 
i'erbaf forms. 

Determine what type of interview 
is'to be conducted and stnjcture it 
to eUcit relevant information. 



Prepare interviewers, mterview 
schedules* provide for space, 
time, and special recording 
equipment, ' 

Use of checklists, rating scales. 
a.id Written reports. 



Recording can be electronic and 
wrttten. Feedback gtven in 
postinterview. 

Tape rccort^ngs or addiiiona! 
observer can be used, in addition 
to examination of written repotts 
arxl empirkia! study. 



Pfcduct Assessment 

Should be de^'ned by specific 
examples. 



Provide guidelines for choosing 
the most relevant and 
representative products. 



Prepare judges by providing them 
with clear speaftcatlon^ and 
examfiles. 



Porfomtance Assissment 

Should be given <n behavioral 
terms. 



Choose the most relevant 
performar>ce m'eihod and adapt to 
specific criteria. 



Prepare judges, provide for space, 
time, and any special equtpment 
needed. 



.Rating scales and wntten reports. Checklists, rating scales, and 
^ written reports. 



Products themselves serve as a 
record. Also written records. 
Feedback written or oral. 

Informal fnter-judge agreement, 
written reports, and cmpirtcat 
study. 



Recordmg can be electronic and 
written. Feedback can be written 
or oral. 

Intemat consistency of the tasks 
and electronic recorcfing are some 
posstbilitfes, in addition to written 
reports and empirical study. 



Written Materia! ' . 

Shoukj be related to objectivt.i 

ot)servabile aspects of writihgoj 



For requested wiitten mattfftal, 
specific guides shouki be provk 
for the student. For praviputly 
produced material, guidelifm ; 
should be given for the numbw 
amount o!. material to.be 
submitted. 

Prepare fudges by prcjylding ttK 
with clear {nstruc^'ons as to wti 
they will t)e judging. 



Rating scales and written rapo 



Recording is written only. 
Feedback can be given in writ 
or oral form. 

tnfonnal Inter-judge agreemen 
examination of wri^en reports, 
and empirical study. 
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rtquire ralatlya^.large samples for results to be meaningful, and it is imfortunately 
Irut of iTK^.dAEL institutions that the required sanipte sizes cannot normally be 
^bbibriiii^.^^S^^ on the resources of several institutions cart 

jbyefcof^ "number8":problem and provide somie valid answers to questions 
aboui aM«^ In order to.do this, however, common pfooadures must 

'^'biiawl^ cooperation must be obtained, and a host 

ci'Mf^k prdbiems must be; c<xif rented and overcome. The difficulty of such an 
uhdbit^k^ is what makes the.CAEL validation studies^ such a unique 9hd mean- 

- iiflfuraco^^ drew on the cooperative efforts of 24 institu- 

tions;, aJMnvbh^ in assessment of hontraJltional learning. The results of thes^ 
^^IK^ the first: timoi "base rate" information on the reliability and 

vaidfiy .d assessme of experiential leapng and, as such, would be an ideal' 
, ^Mfting point for anyone involved in evaluation of assessment at the local level. 




"Warren W< W*nQham and Associates, The CAtL Vabdatan Poport. Princeton. H.J. CAEU Educational Testing Service. 
1970. 
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GAEL HANDBOOKS AND GAEL STUDENT GUIDES 



V 

\ 



Teaching and Assessing Interpersonal dompetence-^ A CAEL $6.00 
Handbook 



Assessing Prior Learning— A GAEL Handbook $6.00 

College-Sponsored Experiential Learning— A CAEL Handbook $6.00 

Expert Assessment of Experiential Learning— A GAEL Handbook $6.00 

Q 

Assessing Occupational Competences— A CAEL Handbook $6.00 

Learning and Assessing interpersonal Competence— A GAEL $3.00 
Student Guide 

Assessing Prior Learning'— A GAEL Student Guide $3.00 

College-Sponsored Experiential Learning— A GAEL Student Guide $3.00 

A Hst of other CAEL publications h available upon request. 



